DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 3, 7, 9, 13, 15, 25, and 27 have been amended. Claims 1-18 and 25-30 are pending in the application.
Response to Arguments
Applicant's arguments filed 7/23/2025 have been fully considered but they are not persuasive. Applicant primarily argues that neither Wang (US 2023/0123820) nor Diaz-Arias (US 2022/0386942), reasonably discloses the newly amended/added limitation of
"wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions”. In explaining such, Applicant discusses the relevant portions of both Wang and Diaz-Arias – and concludes that neither of these Prior arts reasonably disclose or suggest the above mentioned newly added limitation.
Examiner disagrees with the arguments made by the applicant and conclusions drawn therefrom. Specifically, Examiner disagrees with Applicant’s arguments laid forth in pages 9-10 of the remarks of 7/23/2025, where Applicant argues,
Diaz-Arias does not remedy the above deficiencies of Wang. For instance, the cited portions of Diaz-Arias, paragraphs [0029], [0050], and [0051], merely discuss how "to produce a linear acceleration value and an angular acceleration value for each limb from the set of limbs of the at least one subject,""determine a mass value and a torque inertia value for each limb from the set of limbs," and "torque values from the set of limbs." No mention is then made of using these various values to generate images of a character performing actions according to these values. This is because Diaz-Arias is a technique where a "3D representation of the skeleton" is generated from "a set of images of a subject (e.g., an individual performing a physical exercise)" to produce "a risk assessment report" that "can specifically indicate a likelihood of a particular joint being at risk of injury and/or fatigue." Id at paragraph [0068]. In Diaz-Arias, the images of the motion already exist.
[Underlining added by Examiner]
Examiner disagrees with Applicant’s conclusion that, “No mention is then made of using these various values to generate images of a character performing actions according to these values”.
In ¶0028-0029 of Diaz-Arias discloses,
The second machine learning model 123 can include a second set of model parameters (e.g., nodes, weights, biases, etc.) that can be used to determine a set of limbs of the subject(s) based on the set of joints and the set of images. A set of three-dimensional (3D) representations of a skeleton can be generated based on the set of joints and the set of limbs, as described in further detail herein…. The skeleton representation analyzer 124 can perform numerical differentiation on the set of 3D representations of the skeleton of the at least one subject to produce a linear acceleration value and an angular acceleration value for each limb from the set of limbs of the at least one subject. The skeleton representation analyzer 124 can determine a mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject and the 3D representation of the skeleton. Furthermore, in steps 205-207 it is clear that mass and torque inertia is calculated for each limb, and these are physical capabilities of the digital character itself. In fig. 3 and ¶0068 Diaz-Arias discloses,
“In addition, a 3D representation of a skeleton 340 of the subject can be produced by the 3D skeleton reconstruction model. At 305, the 3D representation of the skeleton 340 can be used to compute and analyze physical activity metric (e.g., velocity values, torque values, etc.), as described above.”
Indicating that mass and torques of the limbs are used to generate animated images of a character performing actions according to these values. Skeletal motion model is also seen clearly in fig. 5, 520, 530, e.g., [¶0070].
Therefore, based on the arguments furnished above, Examiner contends that, the digital character of Diaz-Arias does indeed disclose that the mass, inertia and torques are properties of the limbs of the character. And these properties are use not only to find the “risk assessment report”, they are used to animate the digital character as well as seen from figs. 3 [step 304, skeletal motion of 340], and 5 [520, 530 etc.]. For further details see the rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-9, 11-15, 17-18, 25-27, 30 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20230123820 A1, hereinafter Wang), in view of Diaz-Arias et al. (US 20220386942 A1, hereinafter Diaz-Arias).
Regarding claim 1, Wang discloses a processor (1502, fig. 15. Also see abstract), comprising:
one or more circuits to use one or more neural networks to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more nodes of a character graph, wherein an individual node for a given character graph of the one or more characters represents information indicating: The present disclosure relates to systems, non-transitory computer-readable media, and method that utilize a character animation neural network informed by motion and pose signatures to generate a digital video through person-specific appearance modeling and motion retargeting. In particular embodiments, the disclosed systems implement a character animation neural network that includes a pose embedding model to encode a pose signature into spatial pose features. The character animation neural network further includes a motion embedding model to encode a motion signature into motion features. In some embodiments, the disclosed systems utilize the motion features to refine per-frame pose features and improve temporal coherency. In certain implementations, the disclosed systems also utilize the motion features to demodulate neural network weights used to generate an image frame of a character in motion based on the refined pose features, abstract.
For example, a digital video includes a video of a dancer dancing on the sidewalk, a gymnast performing a routine, a coach performing a weightlifting exercise, or a tennis athlete performing a forehand stroke. A digital video can include augmented reality or virtual reality animations, ¶0050.
Moreover, FIG. 12 shows that the digital video generation system 106 can flexibly train the character animation neural network 208 on different target characters to perform motion retargeting, ¶0123.
The digital video generation system 106 then retargets the character to perform a dance sequence including the digital poses 1202, 1204 as illustrated in the image frames 1206, 1208, ¶0124
To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source), ¶0127.
For limitation one or more graph nodes comprising information indicating the one or more actions - see the following explanations.
As shown in FIG. 2, the digital video generation system 106 uses a sequence of digital poses 206 based on one or more motion sources to generate synthesized images 214 for a digital video. As used herein, a digital pose (or pose) refers to digital representation of a character (e.g., an animated or human actor, object, or animal). In particular embodiments, a digital pose includes a structural mapping of joints, limbs, eyes, mouth, torso, or other features or portions of a character.
Thus, according to ¶0047 the limitation, one or more graph nodes (joints) comprising information (structural mappings of joints yielding the digital pose of a digital character) indicating the one or more actions (animated human actor) – is sufficiently disclosed in Wang. Other portions of Wang further solidifies that the teaching is found therein. E.g. in ¶0020, Wang discloses,
For example, in some embodiments, when learning parameters of the character animation neural network, the digital video generation system identifies the sequence of digital poses from a digital video portraying a character in the wild (e.g., a real-world environment such as a dance routine from an online video sharing platform or a social media platform). The digital video generation system can utilize these digital poses to learn the dynamic appearance of the character. Moreover, on some embodiments, to retarget the character to a new motion sequence, the digital video generation system identifies a sequence of digital poses from one or more synthetic motion sources or a digital video portraying another character. Thus, the digital video generation system can extract poses of a variety of characters from a variety of motion sources.
Further support is found in ¶0019, ¶0023-0025, ¶0033, ¶0040, ¶0051-0055, ¶0068, ¶073 … etc.),
Wang is not found disclosing expressly the limitation of, one or more characters represents information indicating: one or more physical capabilities for the one or more characters, and wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions.
However, Diaz-Arias discloses that each character graph including a connected set of nodes representing body components of a respective character, wherein the characters representation [e.g., skeleton] indicates mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject in the 3D representation of the skeleton (The method can further include generating three-dimensional (3D) representations of a skeleton based on the joints and the limbs, Abstract) having corresponding parameter values for the one or more physical capabilities (The second machine learning model 123 can include a second set of model parameters (e.g., nodes, weights, biases, etc.) that can be used to determine a set of limbs of the subject(s) based on the set of joints and the set of images, ¶0028.
The skeleton representation analyzer 124 can perform numerical differentiation on the set of 3D representations of the skeleton of the at least one subject to produce a linear acceleration value and an angular acceleration value for each limb from the set of limbs of the at least one subject. The skeleton representation analyzer 124 can determine a mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject and the 3D representation of the skeleton. The skeleton representation analyzer 124 can further determine a set of torque values from the set of limbs, based on at least one of the mass value and the linear acceleration value, or the torque inertia and the angular acceleration value, ¶0029.
The skeleton representation analyzer 124 of the musculo-skeletal rehabilitation device 110 can determine a load acting on a joint from the set of joints of the 3D representations of the skeleton at a given time. Using load, a set of torque values can be calculated, which can indicate the net result of all muscular, ligament, frictional, gravitational, inertial, and reaction forces acting on the set of joints. To determine/compute a static load on the back joint (e.g., joint L5/S1 shown in FIG. 6) the skeleton representation analyzer 124 can individually compute the torque of inertia of the torso, arms, hands, and handheld object about the back joint using the following equation
torgque=L*W+M*A+I*α
where L represents a torque arm, W represents a weight of a limb from the set of limbs, M represents a mass of the limb, A represents a linear acceleration value of a center of mass of the limb, I represents a torque inertia, and α represents an angular acceleration value of the limb with respect to the ground plane., ¶0051).
Diaz-Arias further discloses in ¶0028-0029 that, the second machine learning model 123 can include a second set of model parameters (e.g., nodes, weights, biases, etc.) that can be used to determine a set of limbs of the subject(s) based on the set of joints and the set of images. A set of three-dimensional (3D) representations of a skeleton can be generated based on the set of joints and the set of limbs, as described in further detail herein…. The skeleton representation analyzer 124 can perform numerical differentiation on the set of 3D representations of the skeleton of the at least one subject to produce a linear acceleration value and an angular acceleration value for each limb from the set of limbs of the at least one subject. The skeleton representation analyzer 124 can determine a mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject and the 3D representation of the skeleton. Furthermore, in steps 205-207 it is clear that mass and torque inertia is calculated for each limb, and these are physical capabilities of the digital character itself. In fig. 3 and ¶0068 Diaz-Arias discloses,
“In addition, a 3D representation of a skeleton 340 of the subject can be produced by the 3D skeleton reconstruction model. At 305, the 3D representation of the skeleton 340 can be used to compute and analyze physical activity metric (e.g., velocity values, torque values, etc.), as described above.”
Indicating that mass and torques of the limbs are used to generate images of a character performing actions according to these values. Skeletal motion model is also seen clearly in fig. 5, 520, 530, e.g., [¶0070].
Therefore, the digital character of Diaz-Arias does indeed disclose that the mass, inertia and torques are properties of the limbs of the character. And these properties are used not only to find the “risk assessment report”, they are used to animate the digital character as well as seen from figs. 3 [step 304, skeletal motion of 340], and 5 [520, 530 etc.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Wang, such that actor's keypoints or anatomical landmarks such as skeleton (understood as character graph) is implemented using Diaz-Arias’s approach such that a specific subject’s skeleton having corresponding parameter values to produce linear and angular acceleration values of each limb (¶0029), modeling a maximum torque of each joint (¶0050), to obtain, one or more characters represents information indicating: one or more physical capabilities for the one or more characters, and wherein the one or more actions depicted in the one or more images are performed according to the one or more physical capabilities of the one or more characters to perform the one or more actions, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious. Furthermore, the combination would enhance the versatility and efficiency of the overall system (see Diaz, ¶0020, In addition, the apparatus and methods described herein can have the advantage of measuring the ROM in substantially real-time (e.g., in less than a second) and changes in muscle strength from multiple joints at the same time and with high accuracy. Furthermore, participants do not have to wear sensors or special pieces of equipment or cloth to use the apparatus and methods described herein.).
Regarding claim 2, Wang in view of Diaz-Arias discloses the processor of claim 1, wherein the given character graph including a connected set of nodes representing body components the one or more characters having corresponding parameter values for the one or more physical capabilities (Diaz-Arias: The method can further include generating three-dimensional (3D) representations of a skeleton based on the joints and the limbs, Abstract) having corresponding parameter values for the one or more physical capabilities (The second machine learning model 123 can include a second set of model parameters (e.g., nodes, weights, biases, etc.) that can be used to determine a set of limbs of the subject(s) based on the set of joints and the set of images, ¶0028.
The skeleton representation analyzer 124 can perform numerical differentiation on the set of 3D representations of the skeleton of the at least one subject to produce a linear acceleration value and an angular acceleration value for each limb from the set of limbs of the at least one subject. The skeleton representation analyzer 124 can determine a mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject and the 3D representation of the skeleton. The skeleton representation analyzer 124 can further determine a set of torque values from the set of limbs, based on at least one of the mass value and the linear acceleration value, or the torque inertia and the angular acceleration value, ¶0029.
The skeleton representation analyzer 124 of the musculo-skeletal rehabilitation device 110 can determine a load acting on a joint from the set of joints of the 3D representations of the skeleton at a given time. Using load, a set of torque values can be calculated, which can indicate the net result of all muscular, ligament, frictional, gravitational, inertial, and reaction forces acting on the set of joints. To determine/compute a static load on the back joint (e.g., joint L5/S1 shown in FIG. 6) the skeleton representation analyzer 124 can individually compute the torque of inertia of the torso, arms, hands, and handheld object about the back joint using the following equation
torgque=L*W+M*A+I*α
where L represents a torque arm, W represents a weight of a limb from the set of limbs, M represents a mass of the limb, A represents a linear acceleration value of a center of mass of the limb, I represents a torque inertia, and α represents an angular acceleration value of the limb with respect to the ground plane., ¶0051).
Regarding claim 3, Wang in view of Diaz-Arias discloses the processor of claim 2, wherein the one or more circuits are further to determine motions for the connected set of nodes of the character graph based, at least in part, upon the current state for the individual node and target state state for the individual node (Diaz-Arias: ¶0037-0039. Nodes are understood implemented in joints, character graph is skeleton and the current and target state data are understood implemented in current state in Hungarian maximum matching algorithm, until the maximum matching of the skeleton is implemented using a target state implemented using ‘threshold number of skeletal points’).
Regarding claim 5, Wang in view of Diaz-Arias discloses the processor of claim 2, wherein the one or more circuits are further to generate different character graphs for different characters having different morphologies of bodily structures (Diaz-Arias: The skeleton representation analyzer 124 can determine a mass value and a torque inertia value for each limb from the set of limbs, based on the at least one total mass value for the at least one subject and the 3D representation of the skeleton, ¶0029.
As described above, the skeleton representation analyzer 124 of the musculo-skeletal rehabilitation device 110 can generate a torque value on each joint of the at least one subject to produce a set of torque values, ¶0053).
Regarding claim 6, Wang in view of Diaz-Arias discloses the processor of claim 1, wherein the one or more circuits are further to generate the one or more images in a sequence to produce physics-based animation for the one or more characters (Wang: character animation neural network informed by motion and pose signatures to generate a digital video through person-specific appearance modeling and motion retargeting, abstract).
Regarding claim 7, 13, although wording is different, the material is considered substantively similar to claim 1 discussed above.
Regarding claim 25, Wang discloses an image generation system, comprising: memory for storing network parameters for the one or more neural networks (¶0133, ¶0158, ¶0160, ¶0168).
Regarding rest of claim 25, although wording is different, the material is considered substantively similar to claim 1 discussed above.
Regarding claims 8, 14, 26, although wording is different, the material is considered substantively similar to claim 2 discussed above.
Regarding claims 9, 15, 27, although wording is different, the material is considered substantively similar to claim 3 discussed above.
Regarding claims 11, 17, although wording is different, the material is considered substantively similar to claim 5 discussed above.
Regarding claims 12, 18, 30 although wording is different, the material is considered substantively similar to claim 6 discussed above.
Claims 4, 10, 16, 28-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Diaz-Arias and further in view of Shin et al. (US 20230267317 A1, hereinafter Shin).
Regarding claim 4, Wang in view of Diaz-Arias discloses the processor of claim 3, wherein the one or more circuits are further to utilize one or more graph neural networks to update state information for the connected set of the nodes (Additionally, as shown in FIG. 2, the digital video generation system 106 utilizes a character animation neural network 208 to generate a pose embedding 210 and a motion embedding 212 based on the sequence of digital poses 206. As used herein, a neural network refers to a model that can be tuned (e.g., trained) based on inputs to approximate unknown functions. In particular, a neural network can include a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, a neural network includes one or more machine-learning algorithms. In addition, a neural network can refer to an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data. To illustrate, a neural network can include a convolutional neural network, a recurrent neural network, a generative adversarial neural network, and/or a graph neural network –¶0052).
Although likely implicit, and explicit mention of the limitation – update state information for the connected set of the nodes by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks, is not found within Wang.
However, Shin discloses further details on graph neural network in ¶0005. To be specific, Shin discloses that,
GNN receives a graph composed of nodes and edges, analyzes edges connecting nodes together with node information, vectorizes each node, and places them on an embedding space. And by exchanging messages with neighboring nodes through edges connected between nodes in the graph structure, the state of each node, that is, information about the node is updated to change the position of the vectorized and arranged nodes on the embedding space. GNN basically assumes that the target node and its neighbor nodes are similar to each other, and transmits and aggregates the information of the neighbor nodes connected with edges as a message so that the information of the target node is updated. GNN is an artificial neural network that exhibits very good performance in representing relationships between a plurality of objects, and is currently being applied to various recommendation devices (¶0005).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to implement the missing and undisclosed elements/components and the functions thereof of GNN of Wang according to disclosed GNN architecture of Shin, to obtain, update state information for the connected set of the nodes by exchanging state information between the connected set of the nodes for each of a set of forward passes through the one or more graph neural networks, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious.
Regarding claims 10, 16, 28, although wording is different, the material is considered substantively similar to claim 4 discussed above.
Regarding claim 29, Wang in view of Diaz-Arias and Shin discloses the image generation system of claim 28, wherein the one or more processors are further to generate different character graphs for different characters having different morphologies or bodily structures (Diaz-Arias: figs. 4-5, ¶0069-0070. Also see ¶0050, ¶0072).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NURUN FLORA whose telephone number is (571)272-5742. The examiner can normally be reached M-F 9:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571) 272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NURUN FLORA/Primary Examiner, Art Unit 2619