Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant’s arguments, filed Sept 14, 2026, with respect to how the newly amended claim features differ from the prior art cited in the last office have been fully considered. These arguments are found to be persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in this office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-7, 10-12, 14-15, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shi (NPL Doc, “MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency”).
As per claim 1, Shi teaches the claimed:
1. One or more processors comprising:
one or more circuits (The system of Shi would have to have some type of circuitry present in their computer hardware in order for their system to function and run on a computer-based system as described in the reference): to:
employ an artificial agent to synthesize motion of a character in a simulation (This is shown on the right side in figure 1 where it shows synthesize motion of a character in a simulation), the artificial agent having been generated by:
receiving video data comprising motion corresponding to at least one actor (This is shown on the left side in figure 1 where shows a few frames of the received video data comprising motion corresponding to at least one actor);
reconstructing one or more movements of the at least one actor in the received video data (This is shown in the transition between the left and right side in figure 1 where the right side shows the reconstructed movements of the at least one actor in the received video data. Also, please see the caption in figure 1 where they state: “Given a monocular video of a performer, our approach, MotioNet, reconstructs a complete representation of the motion, consisting of a single symmetric skeleton, and a sequence of global root positions and 3D joint rotations”), the reconstructing comprising:
extracting, from the video data, a series of estimated kinematic poses for a set of frames in the video data (In section 3, 2nd paragraph “In practice, our network learns to map 2D joint positions, extracted from an input video, into two separate components—(i) a single, symmetric, skeleton represented by its bone lengths; (ii) a dynamic, skeleton-independent, sequence of joint rotations, global root positions, and foot contact labels”.
Also, please see in figure 3 where the right bottom side shows “3D poses” extracted from 2D joint positions from video input); and
generating one or more corrected reconstructed movements by applying, via a physics engine, one or more physics-based constraints to the series of estimated kinematic poses to correct for artifacts arising from noise, inaccuracies, or physical inconsistencies in the series of estimated kinematic poses from the video data (In section 3.2, in the paragraph above where equation (10) appears, Shi recites: “Foot Contact Loss. In order to handle foot sliding artifacts, we supervisedly train the network to output ˜ f that predicts whether each foot is in contact with the ground in each frame” and the paragraph below equation (10) where Shi recites “Based on that binary GT vector, f, another loss, which encourages the velocity of foot positions to be zero during contact frames, is added”.
In these passages, a physics-based constraint arising from inaccuracies or physical inconsistencies is enforced in the estimated kinematic poses from the input video data. The correction that is generated to the reconstructed movements is the loss function that encourages the velocity of foot positions to be zero during contact frames. While Shi does not use the actual words “physics engine”, it is suggested by Shi because Shi has a system that is enforcing the physics-based constraints and applying corrections. Thus, this system is Shi is acting in the same manner as a “physics engine”.
Also, please the end of section 5.2 (in the paragraph titled “Foot Contact Loss”) which applies that the foot sliding corrections for artifacts arising from inaccuracies or physical inconsistencies in the estimated kinematic poses from the input video data as well), the physics-based constraints enforcing at least one of: a rigid-body dynamic constraint, a joint configuration constraint, a joint force or torque constraint, a contact or collision constraint, or a stability or balance constraint (In section 3.2, in the paragraphs above and below where equation (10) is located. These portions teach of physics-based constraints to prevent foot sliding by specifying a constraint that the foot remains stationary when in contact with the ground. This constraint is a rigid-body dynamic constraint, contact constraint, and stability constraint); and
updating, using machine learning and based at least on the one or more corrected reconstructed movements (In section 3.2, in the paragraphs above and below where equation (10) is located. These portions state that a loss function is used to prevent foot sliding by specifying a constraint that the foot remains stationary when in contact with the ground. For example, the paragraph above equation (10) recites: “Foot Contact Loss. In order to handle foot sliding artifacts, we supervisedly train the network to output ˜ f that predicts whether each foot is in contact with the ground in each frame”.
This portion of section 3.2 relates to the training and loss of their machine learning network. For example, the beginning of section 3.2, in the 1st and 2nd paragraphs recite: “To train our network, we leverage publicly available motion capture datasets described in more detail in Section 3.3. We use a combination of full self-supervised and partial self-supervised training … The loss function that we use consists of four components: skeleton loss LS, global root position loss LR, joint position loss LP, and rotations GAN loss LQ_GAN. These components are described in more detail below.”
Also, please see in section 1, 5th paragraph where they refer to: “Instead of inferring a sequence of 3D joint positions, our network learns to extract a sequence of 3D joint rotations applied to a single 3D skeleton. Thus, IK is effectively integrated within the network, and, consequently, is data driven (learned).”), the artificial agent to control one or more motions of the character in the simulation (This is shown in figure 3 on the bottom right portion, e.g. the artificial agent shown as “T-pose” later controls the motions of the character in the simulation shown as “3D poses”. This is done using the forward kinematics module “FK”.
Also, please see figure 1 on the right side where the series of frames shows that the artificial agent is controlled to perform motions that mimic the actor in the video (shown on the left side of figure 1)).
As per claim 4, Shi teaches the claimed:
4. The one or more processors of claim 1, wherein the artificial agent is generated without using motion capture data (On page 3, 2nd column, 1st full paragraph in that column, which recites “… In contrast, our network directly extracts 3D joint rotations from 2D joint positions, producing the same format as that of commercial motion capture systems.” According to this passage, the extracted joint rotations (as part of the artificial agent generation) are extracted from 2D joint positions in the video rather than using a commercial motion capture system. Thus, motion capture system data is not used by rather instead the 2D video is used for extracting joint rotations. Figure 1 on the right side shows the resulting artificial agent that is generated).
As per claim 5, Shi teaches the claimed:
5. The one or more processors of claim 1, wherein applying the one or more physics-based constraint correct for artifacts in the series of estimated kinematic poses, wherein the artifacts include at least one of jitter, foot skating, or ground penetration (In section 3.2, in the paragraph above where equation (10) appears, Shi recites: “Foot Contact Loss. In order to handle foot sliding artifacts, we supervisedly train the network to output ˜ f that predicts whether each foot is in contact with the ground in each frame” and the paragraph below equation (10) where Shi recites “Based on that binary GT vector, f, another loss, which encourages the velocity of foot positions to be zero during contact frames, is added”. In this passage, the foot sliding corresponds to the claimed “foot skating”).
As per claim 6, Shi teaches the claimed:
6. The one or more processors of claim 1, wherein the simulation corresponds to at least one of: an interactive game; or a simulated environment or scene that includes the simulated character (In figure 1 on the right side where it shows the simulation corresponding to a simulated environment or scene that also includes the simulated character).
As per claim 7, Shi teaches the claimed:
7. The one or more processors of claim 1, wherein the simulated character includes at least one of: a participant of a sport; a participant in a performance; or a participant of an activity (In figure 1 on the right side where it shows the simulation character includes a participant in a performance; or a participant of an activity. Figure 8 later shows another example where the simulated character includes a participant of a sport as well).
As per claim 10, Shi teaches the claimed:
10. The one or more processors of claim 1, wherein the reconstructing the movements provides joint rotation data. ( On page 6 in the 1st column in the upper portion where they refer to: “Adversarial Rotation Loss. Applying a loss (only) over the skeleton and joint positions trains the network to reconstruct correct positions”. Also, later on in that same column: in the paragraph below where equation (7) appears, where they refer to: “… denotes the rotations of the jth joint of the output”. Figure 1 shows that over a series of frames the reconstruction is part of movements of the simulated character).
As per claim 11, Shi teaches the claimed:
11. The one or more processors of claim 1, wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations (Shi would have to have some type of computer-based system for performing the simulation operations as shown in figure 1 on the right side and as shown in figure 8 as well); a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations (Shi at the beginning of the abstract: “We introduce MotioNet, a deep neural network that directly reconstructs the motion of a 3D human skeleton from a monocular video”); a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data (Figure 1 on the right side and figure 8 show that synthetic data of a character moving in an animation is generated); a system implemented using at least one language model; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
As per claims 12, 14-15, and 18, these claims are similar in scope to limitations recited in claims 1, 4-5, and 10 respectively, and thus are rejected under the same rationale. Shi in figure 8 shows an example of a received video that comprises motion by at least one actor interacting with at least one object (e.g. an actor standing on or jumping over objects on the ground or the actor holding ski poles).
As per claim 20, this claim is similar in scope to limitations recited in claims 1 and 12, and thus is rejected under the same rationale.
Claims 2 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of Grigorev (NPL Doc, “StylePeople: A Generative Model of Fullbody Human Avatars”).
As per claim 2, Shi alone does not explicitly teach the claimed limitations.
However, Shi in combination with Grigorev teaches the claimed:
2. The one or more processors of claim 1, wherein the artificial agent comprises a generative machine learning model (Grigorev in the abstract “We then propose a generative model for such avatars that can be trained from datasets of images and videos of people. The generative model allows us to sample random avatars as well as to create dressed avatars of people from one or few images”. In this instance, the artificial agent (the avatar) has a generative model machine learning model is order to create dressed avatars).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the artificial agent to comprise a generative machine learning model as taught by Grigorev with the system of Shi in order to help create virtual avatar models who are dressed like the sample images and videos of people provided in the training set. Shi also suggests that it would be advantageous in the future to use a generative model with their system at the end of their paper, e.g. Shi on page 13 towards the top of the 1st col, Shi recites “In other possible future works, we are interested in … creating a generative model, where novel motion styles can be sampled.” (the paragraph before Appendix A appears).
As per claim 13, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
Claims 3, 8-9, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of Peng et al. (NPL Doc, “SFV: Reinforcement Learning of Physical Skills from Videos”).
As per claim 3, Shi alone does not explicitly teach the claimed limitations.
However, Shi in combination with Peng teaches the claimed:
3. The one or more processors of claim 1, wherein the video data comprises monocular broadcast video (Shi in the abstract refers to using monocular video but not “broadcast” monocular video per se. Peng teaches this feature, e.g. please see Peng in the abstract “… Our approach, based on deep pose estimation and deep reinforcement learning, allows data-driven animation to leverage the abundance of publicly available video clips from the web, such as those from YouTube”. In this instance, video data from YouTube is monocular broadcast video because it is video from a single eyepoint perspective (monocular) and it is broadcast to many people across the internet using the YouTube service).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the video data to comprise monocular broadcast video as taught by Peng with the system of Shi because broadcast video from services such as Youtube is widely available and offers a large variety of different topics of video.
As per claim 8, Shi teaches the claimed:
8. The one or more processors of claim 1, wherein -- the simulated character is a virtual participant of the at least one of: the one or more sporting events, the one or more performances, or the one or more activities (Please see Shi in figures 1 and 8 where it shows the simulated character participating in a sport, performance, or activity).
Shi alone does not explicitly teach the remaining claim limitations relating to “the video data comprises broadcasted events” per se.
However, Shi in combination with Peng teaches the claimed:
wherein the video data comprises broadcasted events (Peng in the abstract “… Our approach, based on deep pose estimation and deep reinforcement learning, allows data-driven animation to leverage the abundance of publicly available video clips from the web, such as those from YouTube”. In this instance, video data from YouTube is broadcast to many people across the internet using the YouTube service) of at least one of: one or more sporting events, one or more performances, or one or more activities, wherein the at least one actor is at least one human participant of the at least one of: the one or more sporting events, the one or more performances, or the one or more activities (Peng in figure 4 on the left side shows the broadcast video and actor as at least one human participant of a sporting event. Peng in figures 3-5 show broadcast videos and an actor as at least one human participant of a performance/ activity).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the video data to comprise broadcast events as taught by Peng with the system of Shi. The motivation of claim 3 is incorporated herein.
As per claim 9, Shi teaches the claimed:
9. The one or more processors of claim 1, wherein the video data comprises motion by the at least one actor interacting with at least one object (Shi in figure 8 shows an example of a received video that comprises motion by at least one actor interacting with at least one object (e.g. an actor standing on or jumping over objects on the ground or the actor holding ski poles).
Shi alone does not explicitly teach the remaining claim limitations.
However, Shi in combination with Peng teaches the claimed:
and the one or more circuits are further to employ the artificial agent to synthesize motion of one or more simulated objects with which the simulated character interacts (This is shown in Peng figures 7 and 8 where the artificial agent to synthesize motion of one or more simulated objects (a virtual box) with which the simulated character interacts).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to simulate character interaction with one or more simulated objects as taught by Peng with the system of Shi in order to generate a wider range of different scenarios and body motions in the output simulation video, e.g. many sports or performances typically involve an actor interacting with an object.
As per claims 16 and 17, these claims are similar in scope to limitations recited in claims 8 and 9 respectively, and thus are rejected under the same rationale.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Shi in view of Pile et al. (Patent No. US 9522330 B2).
As per claim 19, Shi alone does not explicitly teach the claimed limitations.
However, Shi in combination with Pile teaches the claimed:
19. The one or more processors of claim 12, wherein the one or more circuits are to further use the artificial agent as a controller in an interactive game (Please see Pile in figures 1A-B and figure 2 where it shows that the movement of the recorded person that includes the avatar on screen (artificial agent) is used as a controller in an interactive game. Also, please see Pile in col 3, lines 58-63 “FIG. 1A illustrates a scenario in which tracking device 20 tracks game player 18 such that the movements of game player 18 may be interpreted by gaming system 12. In particular, the movements of game player 18 are interpreted by depth analysis system 10 so as to effect control over video game 16 provided by gaming system 12 “ and Pile in col 9, lines 27-31 “At 56, FIG. 2 shows display 14 visually presenting avatar 24. In some embodiments, virtual skeleton 44 may be used to render avatar 24, and, since virtual skeleton 44 changes poses as human subject 18 changes poses, avatar 24 may mimic the movements of human subject 18”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further use the artificial agent as a controller in an interactive game as taught by Pile with the system of Shi in order to incorporate the avatar created by Shi into an interactive application or interactive and entertaining experience. Using the artificial agent as a controller in an interactive game may provide a more natural way to provide input or control into the system to simulate the avatar’s movement (such as a game).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL F HAJNIK whose telephone number is (571) 272-7642. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights