DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1, 11 and 20 are rejected under 35 U.S.C. 103 as being unpatentable Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang)
Regarding independent claim 1, Wang teaches a method comprising:
acquiring a plurality of images (see at least [0049] In one or more embodiments, the digital video generation system 106 extracts the sequence of digital poses 206 based on a motion source that includes synthetic poses 202. As used herein, a synthetic pose includes representations of a character that are independent of a real-world representation. To illustrate, a synthetic pose includes an artificial pose, a modified pose, or an uncontextualized pose. For example, a synthetic pose includes fake poses or poses that are unassociated with a real-world representation such as an image or video (e.g., due to being abstracted out for privacy or other reasons). As additional or alternative examples, a synthetic pose includes one or more of a user-generated pose, a machine-created pose, a partial pose, a blended pose between multiple poses, etc. In other implementations, a synthetic pose includes a modified pose based on an original pose extracted from a digital image”), a first video ( see at least [0050] as shown in Fig.2 “ In other embodiments, the digital video generation system 106 extracts the sequence of digital poses 206 based on a motion source that includes digital videos 204. As used herein, digital videos refer to a combination of image frames. In particular embodiments, digital videos include a sequential display of a character in motion. Specifically, a digital video can include a plurality of image frames that, when displayed sequentially, portray a character in motion in a real-world or animated environment. For example, a digital video includes a video of a dancer dancing on the sidewalk, a gymnast performing a routine, a coach performing a weightlifting exercise, or a tennis athlete performing a forehand stroke. A digital video can include augmented reality or virtual reality animations.”), wherein the plurality of images indicate visual information of a source object ([0049] In one or more embodiments, the digital video generation system 106 extracts the sequence of digital poses 206 based on a motion source that includes synthetic poses 202. As used herein, a synthetic pose includes representations of a character that are independent of a real-world representation. To illustrate, a synthetic pose includes an artificial pose, a modified pose, or an uncontextualized pose. For example, a synthetic pose includes fake poses or poses that are unassociated with a real-world representation such as an image or video (e.g., due to being abstracted out for privacy or other reasons). As additional or alternative examples, a synthetic pose includes one or more of a user-generated pose, a machine-created pose, a partial pose, a blended pose between multiple poses, etc. In other implementations, a synthetic pose includes a modified pose based on an original pose extracted from a digital image”), and the first video indicates animation of a target object (see at least [0050] as shown in Fig.2 “ In other embodiments, the digital video generation system 106 extracts the sequence of digital poses 206 based on a motion source that includes digital videos 204. As used herein, digital videos refer to a combination of image frames. In particular embodiments, digital videos include a sequential display of a character in motion. Specifically, a digital video can include a plurality of image frames that, when displayed sequentially, portray a character in motion in a real-world or animated environment. For example, a digital video includes a video of a dancer dancing on the sidewalk, a gymnast performing a routine, a coach performing a weightlifting exercise, or a tennis athlete performing a forehand stroke. A digital video can include augmented reality or virtual reality animations.”);
generating a plurality of animation models for the target object based on the first video (see at least [062] In certain embodiments, the digital video generation system 106 generates the pose signatures 302 by combining the dense body mapping images and the keypoint data images. For example, in some embodiments, the digital video generation system 106 concatenates the dense body mapping images and the keypoint data images on a per-frame basis to form a pose signature [AltContent: rect].sub.i ∈ [AltContent: rect].sup.6×W×H for each input frame. The terms W and H represent the RGB image dimensions of the input frame, the dense body mapping images, and the keypoint data images.”; [0089] At an act 404, the digital video generation system 106 generates digital poses of the actor portrayed in the identified motion source. It will be appreciated that a motion source of synthetic poses may already include digital poses or character representations in an abstract form (e.g., keypoint or dense correspondence estimations). However, for a motion source that includes a digital video, the act 404 comprises representing the actor portrayed in image frames via digital poses.[0090] To illustrate, the digital video generation system 106 can use one or more different approaches to generating digital poses of an actor depicted in an image frame of the motion source. In some embodiments, the digital video generation system 106 generates a digital pose by determining a shape, outline, segmentation, or structural approximation of an actor depicted in an image frame of the motion source. In other embodiments, the digital video generation system 106 generates a digital pose by performing object reconstruction (e.g., using depth maps) to reconstruct a three-dimensional surface of the actor based on sampled surface depth points. Still, in other embodiments, the digital video generation system 106 generates a digital pose by generating a three-dimensional representation of the actor using a skinned multi-person linear model.); and
fusing the source object and the plurality of animation models for the target object to generate a second video for the source object (see at least [0028] Upon training, the digital video generation system can then apply the character animation neural network to other input sequences. In particular, the digital video generation system can retarget the character to generate digital videos that portray alternative target sequences. For example, the digital video generation system can identify a new sequence of digital poses and utilize the animation neural network to generate new pose embeddings, new motion embeddings, and ultimately a new digital video portraying the character performing the new sequence of digital poses. In this manner, the digital video generation system can generate frames of a variety of different digital videos and animated sequences (e.g., without retraining the overall model). [0047] As shown in FIG. 2, the digital video generation system 106 uses a sequence of digital poses 206 based on one or more motion sources to generate synthesized images 214 for a digital video. As used herein, a digital pose (or pose) refers to digital representation of a character (e.g., an animated or human actor, object, or animal). In particular embodiments, a digital pose includes a structural mapping of joints, limbs, eyes, mouth, torso, or other features or portions of a character. For example, a digital pose can include a dense body mapping image (e.g., a DensePose representation comprising an RGB (red, green, blue) image indicating correspondences between a two-dimensional input image depicting a character and a three-dimensional surface-based representation of the character). As another example, a digital pose can include a keypoint data image (e.g., an RGB image of an OpenPose representation of a character's anatomical keypoints or body parts based on part affinity fields). In yet another example, a digital pose includes three-dimensional representations of a character as generated by a skinned multi-person linear model. Relatedly, a pose signature can include a combination of digital poses, such as a combination of a dense body mapping image and a keypoint data image.[0085] Additionally, in some embodiments, the digital video generation system 106 trains the character animation neural network 208 based on different types of training inputs than described above. For example, in one or more embodiments, the digital video generation system 106 uses pose and motion signatures based on three-dimensional representations (as opposed to two-dimensional images like dense body mapping images or keypoint data images). An example three-dimensional representation includes skinned multi-person linear models. As another example, the digital video generation system 106 can provide additional training input that comprises a motion of the camera (as opposed to assuming a fixed camera position). In a further example, the digital video generation system 106 provides additional training input or intra-model modifications to increase motion retargeting capabilities. [[0036] In addition, the digital video generation system can also improve system flexibility by generating a wide variety of retargeted digital videos. For example, unlike some conventional video synthesis systems, the digital video generation system can retarget myriad different fully body motions to a character. These full body motions may include complex motions such as dance or gymnastics routines. Also, the digital video generation system can flexibly retarget motion in a way that accurately transfers clothing and accessary animations to a variety of new, complex character motion”; [0127] To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source).”) Wang is understood to be silent on the remaining limitation of claim 1.
In the same field of endeavor, Smolic teaches acquiring a plurality of images, wherein the plurality of images indicate visual information of a source object (see at least [0015] According to a first aspect of the invention, there is provided a method for generating a three-dimensional model, the method comprising the following steps: [0016] (a) providing a plurality of images of a scene captured by a plurality of image capturing devices [0142] In step 101 a plurality of images are provided. The plurality of images are of a scene captured by a plurality of image capturing devices. The scene in this context can be understood as a 3D region with an object, such as a person, located therein. The object may be dynamic, meaning that it is moving between frames of images captured by the plurality of image capturing devices, and the background of the scene may be relatively stationary. [0143] The image capturing devices may be any device capable of capturing static images or dynamic images, e.g. video. The image capturing devices could, for example, be professional video cameras as used in the television and film industries. Such image capturing devices may be dispersed around the scene in a dense, regular, arrangement such that a large amount of information for the scene is captured in the images, and that there is a large amount of overlap between the images captured by the different image capturing devices. The image capturing devices may be simple hand-held cameras, video recorders, or even mobile phones with the capability to capture images and/or record videos. The image capturing devices may be sparsely arranged around the scene such that some aspects of the scene may not be captured in detail, and/or such that there may not be much overlap between the images captured by the image capturing devices. It will be appreciated that not all of the image capturing devices need be the same, and that they could comprise a number of different types of image capturing devices.” .);
generating a three-dimensional model for the source object based on the plurality of images (see at least [0022] The method according to the present invention is able to use a plurality of images of a scene captured by a plurality of image capturing devices to generate a three-dimensional model. In other words, the method is able to generate a three-dimensional model of an object in the scene, or all or part of the scene including the object using the captured image data. The object may be considered as the foreground of the scene”; [0192] By combining the object point cloud and the shape volume, the present invention is able generate three-dimensional models that accurately reflect the object in the scene even in sparse image capturing device setups. The present invention therefore enables the benefits of the object point cloud and the silhouette information techniques to be achieved without the associated disadvantages. The generated three-dimensional model is able to preserve the detail of the object point cloud and the completeness of the estimated three-dimensional shape volume while avoiding inflation of the three-dimensional model)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang with generating a three-dimensional model for the source object based on the plurality of images as seen in Smolic because this modification would use the three-dimensional model in video content creation, and in particular Free-Viewpoint Video (FVV) content creation or the creation of Augmented Reality (AR) and/or Virtual Reality (VR) and/or other Mixed Reality (MR) content ([0023] of Smolic). Wang and Smolic are understood to be silent on the remaining limitation of claim 1.
In the same field of endeavor, Zhang teaches acquiring image ([0073] as shown in Fig.5 “For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown)”) and a first video, wherein the image indicate visual information of a source object, and the first video indicates animation of a target object ([0072] FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502.”);
PNG
media_image1.png
304
424
media_image1.png
Greyscale
generating a three-dimensional model for the source object based on the image ([0073] as shown in Fig.5 “For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown)”;
generating a plurality of animation models for the target object (see at least [0076] At 606, the RGB video may be input to the neural network 104. In at least one embodiment, the circuitry 202 may input the RGB video to the neural network 104. [0077] At 608, a pose of the human object 116 may be estimated in each frame of the RGB video based on an output of the neural network 104 for the input. In at least one embodiment, the circuitry 202 may estimate the pose of the human object 116 in each frame of the RGB video based on the output of the neural network 104 for the input (e.g., the sequence of frames 306A . . . 306N of the RGB video). The estimated pose may include features associated with the human joints 308A . . . 308N and face landmarks 310A . . . 310N of the human object 116. [0078] At 610, from the input human-dynamics sequence, a key-frame may be selected for which the estimated pose is closest to a reference human pose. In at least one embodiment, the circuitry 202 may select, from the input human-dynamics sequence, a key-frame for which the estimated pose is closest to a reference human pose (for example, a T-pose). [0079] At 612, from the selected key-frame and up to a number of frames of the input human-dynamics sequence, a tracking sequence may be generated for a 3D human mesh of the human object 116. In at least one embodiment, the circuitry 202 may generate a tracking sequence, from the selected key-frame and up to a number of frames of the input human-dynamics sequence. The generated tracking sequence may include final values of parameters of articulate motion and non-rigid motion of a set of 3D human points on the 3D human mesh”; [0072] FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502 [0080] At 614, an FVV may be generated based on the generated tracking sequence. In at least one embodiment, the circuitry 202 may generate the FVV based on the generated tracking sequence.); and
fusing the three-dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object, wherein in the second video, the target object in the first video is replaced with the source object(see at least [0046] FIGS. 3A and 3B are diagrams which, collectively, illustrate an exemplary processing pipeline for generation of a tracking sequence of a 3D human mesh to be rendered as a free-viewpoint video [0072] FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502. [0073] For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown). The circuitry 202 may apply the generated tracking sequence for the first 3D human mesh on the textured 3D human mesh 506 of the second human object 504. Once applied, an FVV (e.g., an FVV frame 508) may be rendered to re-enact the performance of the first human object 502 by the second human object 504.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang and generating a three-dimensional model for the source object based on the plurality of images of Smolic with applying the generated tracking sequence for the first 3D human mesh on the textured 3D human mesh of the second human object as seen in Zhang because this modification would re-enact the performance of the first human object by the second human object ([0073] of Zhang).
Thus, the combination of Wang, Smolic and Zhang teaches a method comprising: acquiring a plurality of images and a first video, wherein the plurality of images indicate visual information of a source object from a plurality of perspectives, and the first video indicates animation of a target object; generating a three-dimensional model for the source object based on the plurality of images; generating a plurality of animation models for the target object based on the first video; and fusing the three-dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object, wherein in the second video, the target object in the first video is replaced with the source object.
Regarding independent claim 11, Wang teaches an electronic device , comprising: at least one processor; and a memory coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform actions (see at least [0134] Each of the components of the computing device 1300 can include software, hardware, or both. For example, the components of the computing device 1300 can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device) comprising: Remaining limitations of claim 11 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding independent claim 20, Wang teaches a computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform actions (see at least [0134] Each of the components of the computing device 1300 can include software, hardware, or both. For example, the components of the computing device 1300 can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the digital video generation system 106 can cause the computing device(s) (e.g., the computing device 1300) to perform the methods described herein. Alternatively, the components of the computing device 1300 can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components of the computing device 1300 can include a combination of computer-executable instructions and hardware.;) comprising: Remaining limitations of claim 20 is similar scope to claim 1 and therefore rejected under the same rationale.
2. Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Wang et al, U.S Patent Application Publication No.20240104828 (“Wang_828”)
Regarding claim 2, Wang, Smolic and Zhang teach the method according to claim 1, wherein generating a three-dimensional model for the source object based on the plurality of images comprises:
sampling the plurality of images to obtain a plurality of key image frames (see at least [0150] of Smolic “An example pose estimation operation involves estimating the pose of the plurality of image capturing devices in 3D space, and in particular in the three-dimensional coordinate system of the scene when capturing the plurality of images. [0151] One approach for estimating the pose of the image capturing devices is to use Structure from Motion (SfM) techniques on a frame-by-frame basis. But such an approach could be intractable and computationally expensive.[0152] Another approach for estimating the pose of the image capturing devices is to use monocular Simultaneous Localization and Mapping (SLAM) algorithms. But such approaches may not be successfully due to their dependency on good initialisation and instability for very small motions, which is typically the case with handheld devices. [0153] A beneficial approach is to estimate accurate calibration using SfM at only at a subset of the time intervals. These small subset of frames are denoted as keyframes, and in one example implementation, there is one keyframe for every second of video. In between these time intervals, an algorithm is applied to interpolate calibration parameters for each image capturing device individually.” see at least [0013] of Zhang “The following described implementations may be found in the disclosed system and method for detection-guided tracking of human-dynamics. Exemplary aspects of the disclosure provide a system that may deal with an input human-dynamics sequence containing an RGB video and scanned geometry (which can be either point cloud or a mesh). The system may generate a temporally consistent tracking sequence which provides a parametrization of skeletal joints, articulate motion, nonrigid motion, and the deformation of each 3D point. For each frame of the input human-dynamics sequence, this system may utilize a state-of-the-art neural network to detect 2D human joints and face landmarks and may then fit a human body template to the detected 2D points and the scanned geometry. The system may select a key-frame from the input human-dynamics sequence and from the key-frame up to a number of frames, the system may generate a tracking sequence for a 3D human mesh.; see at least Fig. 1);
determining a sparse point cloud for the target object based on the plurality of key image frames (see at least [0160] of Smolic “ In the second step, when all successful matches have been found for frame s.sub.i(j+1), every feature f.sub.j+1(k) will have a valid match f.sub.j′(k) in frame s.sub.i′(j), which is known to correspond to a 3D point in the reconstruction. The updated correspondences are then used as an input to a PnP algorithm for computing the image capturing device pose for the new frame. Different PnP algorithms may be used depending on the 3D geometry of the scene.”; [0168] In a preferred approach, KAZE features are used to generate the sparse 3D point cloud as the computation can be easily parallelised. Further, the resulting sparse point clouds are slightly denser compared to the SIFT approach. The KAZE approach may not accurately detect feature points in dark areas so it is further preferred to enhance the images prior to using the KAZE features. 169] Referring to FIG. 1, in step 104, the object point cloud is then extracted from the generated 3D point cloud. The object point cloud may be extracted using the silhouette information, otherwise known as object masks, obtained from a segmentation operation performed on the images. This results in segmenting the generated 3D point cloud into the object point cloud, i.e. the foreground point cloud, and a background point cloud.; 0170] Referring to FIGS. 2(a)-(d) there is shown resultant object point clouds 201-204 generated for a first frame (FIGS. 2(a)-(b)) and a second frame (FIGS. 2(c)-2(d)) of a video sequence captured by a plurality of image capturing devices in a sparse setup. The object point clouds 201, 203 in FIGS. 2(a) and 2(c) show the SIFT+PMVS approach, while the object point clouds 202, 204 in FIGS. 2(b) and 2(d) show the preferred KAZE approach with colour enhancement. FIGS. 2(a)-2(d) highlight that the KAZE approach generates denser point clouds 202, 204, and thus enables a more accurate 3D model to be generated.”; see at least [0084] of Zhang “In accordance with an embodiment, the system may further include a scanning setup (such as the scanning setup 108 (FIG. 1)). The scanning setup may be configured to capture the RGB video that includes a sequence of frames of the human object from a set of viewpoints in a 3D environment (such as the 3D environment 114). The scanning setup may be further configured to acquire the geometry information of the human object from the set of viewpoints. The geometry information 304A . . . 304N may include a sequence of point cloud frames of the human object. In accordance with an embodiment, the received geometry information may include a sequence of 3D meshes. Each 3D mesh of the sequence of 3D meshes may include 3D coordinate information, texture information, shading information, and lighting information associated with the human object)
training a first model based on the sparse point cloud for the target object (see at least [0019] of Zhang “The neural network 104 may be a computational network or a system of artificial neurons, arranged in a plurality of layers, as nodes. For example, the neural network 104 may be a multi-stage Convolutional Neural Network (CNN), or a hybrid variant of one or more neural network architecture(s). The plurality of layers of the neural network 104 may include an input layer, one or more hidden layers, and an output layer. Each layer may include one or more nodes (or artificial neurons, represented by circles, for example). Outputs of all nodes in the input layer may be coupled to at least one node of hidden layer(s). Similarly, inputs of each hidden layer may be coupled to outputs of at least one node in other layers of the neural network 104. Outputs of each hidden layer may be coupled to inputs of at least one node in other layers of the neural network 104. Node(s) in the final layer may receive inputs from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer may be determined from hyper-parameters of the neural network 104. Such hyper-parameters may be set before or while training the neural network 104 on a training dataset.”;; and
determining the three-dimensional model using the trained first model based on the plurality of key image frames (see at least [0083] of Zhang” In accordance with an embodiment, the system may further include a memory (such as the memory 204 (FIG. 2)). The memory 204 may be configured to store the neural network pre-trained to detect the human joints and the face landmarks. In accordance with an embodiment, the neural network may be a multi-stage Convolutional Neural Network (CNN).”; [0192] of Smolic “By combining the object point cloud and the shape volume, the present invention is able generate three-dimensional models that accurately reflect the object in the scene even in sparse image capturing device setups. The present invention therefore enables the benefits of the object point cloud and the silhouette information techniques to be achieved without the associated disadvantages. The generated three-dimensional model is able to preserve the detail of the object point cloud and the completeness of the estimated three-dimensional shape volume while avoiding inflation of the three-dimensional model In addition, the same motivation is used as the rejection for claim 1. Wang, Smolic and Zhang are understood to be silent on the remaining limitations of claim 2.
In the same field of endeavor, Wang_828 teaches training a first model according to a preset training condition based on the sparse point cloud for the target object (see at least [0028] FIG. 1 illustrates an overall training process 100 for training the improved NeRF-based model for novel view and unseen pose synthesis of dynamic scenes. It should be noted that the training process 100 here depicts steps performed during a single training iteration. The same steps may be repeated for several iterations until the model is deemed to be sufficiently complete. As an example and not by way of limitation, the training process 100 may be repeated for 30 iterations, where each iteration includes rendering an image from a particular camera viewpoint (e.g., 1 of 30 different camera viewpoints), an RGB-D image associated with that particular camera viewpoint, a window or a sequence of image frames (e.g., 10 RGB-D images), and a set of key/reference frames (e.g., 3 key frames). The RGB-D image that is used for generating the appearance code may be one of the window or sequence of image frames (e.g., 10 RGB-D images) or it may be a different one. In one example implementation, the model discussed herein is trained for 30 different camera viewpoints.” [0033] Since the query pose 104 is now represented in the 3D space, a network may be needed to extract features from such 3D space. As depicted, a 3D backbone or a sparse convolution neural network 106 (also interchangeably herein referred to as SparseConvNet) may be used to extract the features from the query pose 104 and generate a 3D feature volume 108. The 3D feature volume 108 may include feature vectors corresponding to the features extracted from the query pose 104 using the 3D backbone 106. In some embodiments, to take advantage of the rich semantic and detailed cues from images, a 2D convolution network (e.g., ResNet34) may be used to encode the image feature map E(I.sub.t) for the given image I.sub.t. Specifically, features may be extracted from the ResNet34, and then three convolutional layers may be utilized to reduce the dimension followed by a SparseConvNet to encode the features anchored to the sparse point clouds.”) and
determining the three-dimensional model using the trained first model based on the plurality of key image frames (see at least [0033] Since the query pose 104 is now represented in the 3D space, a network may be needed to extract features from such 3D space. As depicted, a 3D backbone or a sparse convolution neural network 106 (also interchangeably herein referred to as SparseConvNet) may be used to extract the features from the query pose 104 and generate a 3D feature volume 108. The 3D feature volume 108 may include feature vectors corresponding to the features extracted from the query pose 104 using the 3D backbone 106. In some embodiments, to take advantage of the rich semantic and detailed cues from images, a 2D convolution network (e.g., ResNet34) may be used to encode the image feature map E(I.sub.t) for the given image I.sub.t. Specifically, features may be extracted from the ResNet34, and then three convolutional layers may be utilized to reduce the dimension followed by a SparseConvNet to encode the features anchored to the sparse point clouds. 4] To obtain a subset of features corresponding to one or more points of interest of the dynamic scene and encode this subset of features into the appearance code 120, camera rays may be cast or shoot from a particular camera point or query point 110 into the 3D feature volume 108. The subset of features may be extracted based on the camera rays hitting at several points/locations in the 3D feature volume 108. Using the subset of features extracted from the 3D feature volume 108, another neural network (e.g., encoder) may be used to encode the subset of features into the appearance code or first latent representation 120. In some embodiments, to obtain the appearance code for each point sampled along the camera ray, a trilinear interpolation may be utilized to query the code at the continuous 3D locations. ψ(x.sup.i.sub.t,, E) is adopted to represent the appearance code for point x.sup.i.sub.t. The appearance code 120 together with a pose code (e.g., pose code 120) may be forwarded into a neural network (e.g., density and color model 170) to predict a density and color per pixel of an image to render, as discussed in further detail below. The appearance code learned on each single frame may model the details on the human body and help recover some missing pixels in the 3D space. For instance, the appearance code 120 encodes appearance information or fine-level details of one or more objects in a dynamic scene. The dynamic scene may include the one or more objects in motion. For example, if the dynamic scene includes a person in motion, then the appearance code encodes facial characteristics of the person, body characteristics of the person, cloth wrinkles, details of clothes that the person is wearing, etc.”)
Therefore, in combination of Wang, Smolic and Zhang, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a three-dimensional model for the source object based on the plurality of images of Smolic with training a first model to a training condition as seem in Wang_828 because this modification would be trained for 30 different camera viewpoints ([0028] of Wang_828).
Thus, the combination of Wang, Smolic and Zhang and Wang_828 teaches wherein generating a three-dimensional model for the source object based on the plurality of images comprises: sampling the plurality of images to obtain a plurality of key image frames; determining a sparse point cloud for the target object based on the plurality of key image frames; training a first model according to a preset training condition based on the sparse point cloud for the target object; and determining the three-dimensional model using the trained first model based on the plurality of key image frames.
Regarding claim 12, Wang, Smolic and Zhang teach the electronic device according to claim 11, Remaining limitations of claim 12 is similar scope to claim 2 and therefore rejected under the same rationale.
3. Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Wang et al, U.S Patent Application Publication No.20240104828 (“Wang_828”) further in view of XU et al, U.S Patent Application Publication No.20210082135 (“XU”)
Regarding claim 3, Wang, Smolic and Zhang and Wang_828 teach the method according to claim 2, wherein the preset training condition comprises: iteratively training the first model for a first preset number of times based on the sparse point cloud of the target object (see at least [0028] of Wang_828 “ FIG. 1 illustrates an overall training process 100 for training the improved NeRF-based model for novel view and unseen pose synthesis of dynamic scenes. It should be noted that the training process 100 here depicts steps performed during a single training iteration. The same steps may be repeated for several iterations until the model is deemed to be sufficiently complete. As an example and not by way of limitation, the training process 100 may be repeated for 30 iterations, where each iteration includes rendering an image from a particular camera viewpoint (e.g., 1 of 30 different camera viewpoints), an RGB-D image associated with that particular camera viewpoint, a window or a sequence of image frames (e.g., 10 RGB-D images), and a set of key/reference frames (e.g., 3 key frames). The RGB-D image that is used for generating the appearance code may be one of the window or sequence of image frames (e.g., 10 RGB-D images) or it may be a different one. In one example implementation, the model discussed herein is trained for 30 different camera viewpoints.” [0033] Since the query pose 104 is now represented in the 3D space, a network may be needed to extract features from such 3D space. As depicted, a 3D backbone or a sparse convolution neural network 106 (also interchangeably herein referred to as SparseConvNet) may be used to extract the features from the query pose 104 and generate a 3D feature volume 108. The 3D feature volume 108 may include feature vectors corresponding to the features extracted from the query pose 104 using the 3D backbone 106. In some embodiments, to take advantage of the rich semantic and detailed cues from images, a 2D convolution network (e.g., ResNet34) may be used to encode the image feature map E(I.sub.t) for the given image I.sub.t. Specifically, features may be extracted from the ResNet34, and then three convolutional layers may be utilized to reduce the dimension followed by a SparseConvNet to encode the features anchored to the sparse point clouds.”) In addition, the same motivation is used as the rejection for claim 2. Wang, Smolic and Zhang and Wang_828 are understood to be silent on the remaining limitations of claim 3.
In the same field of endeavor, XU teaches wherein the preset training condition comprises: iteratively training the first model for a first preset number of times based on the sparse point cloud of the target object (see at least [0048] Aiming at the problems of the abovementioned depth completion method, the embodiments of the disclosure propose that a to-be-diffused map is obtained at first based on a collected sparse depth image and a 2D image of a 3D scenario and then pixel-level diffusion is implemented on the to-be-diffused map to obtain a completed depth image, so that each piece of sparse point cloud data in the sparse depth image is fully utilized, and a more accurate completed depth image is obtained.” [0219] In S105, the completed depth image is determined as a to-be-diffused map, and the operation that the diffusion intensity of each pixel in the to-be-diffused map is determined based on the to-be-diffused map and the feature map, the operation that the diffused pixel value of each pixel in the to-be-diffused map is determined based on the pixel value of each pixel in the to-be-diffused map and the diffusion intensity of each pixel in the to-be-diffused map and the operation that the completed depth image is determined based on the diffused pixel value of each pixel in the to-be-diffused map are repeatedly executed until a preset repetition times is reached”) and
reducing a learning rate of the first model in response to iterative training being performed for a second preset number of times, wherein the second preset number of times is less than the first preset number of times (see at least [0221] In some embodiments of the disclosure, the preset repetition times may be set to be eight. After the completed depth image is obtained, the abovementioned operations may be continued to be executed for seven times for a completed depth image to implement more complete pixel diffusion. It is to be noted that the preset repetition times may be set according to a practical requirement. No limits are made in the embodiments of the disclosure.”; [0227] Exemplarily, the impact of a value of the preset repetition times on an error of the completed depth image is presented in an embodiment of the disclosure. As shown in FIG. 12A, a KITTI dataset is adopted for testing, the abscissa is the preset repetition times, and the ordinate is a Root Mean Square Error (RMSE), a unit of the RMSE being mm. The three curves in the figure are results obtained when different values are adopted for an all-sample test number (epoch) respectively. It can be seen from FIG. 12A that: when epoch=10, namely all samples in the KITTI dataset are tested for 10 times, the RMSE decreases along with increase of the preset repetition times. When the preset repetition times is 20, the RMSE is minimum, close to 0; when epoch=20, the RMSE decreases at first along with the preset count repeat and then is kept unchanged, and the RMSE is close to 0; and when epoch=30, the RMSE decreases along with increase of the preset repetition times and then increases to a low extent with a maximum of the RMSE not more than 5 until the RMSE is finally close to 0. FIG. 12B is a diagram of testing results obtained by an NYU dataset. Like FIG. 12A, the abscissa is the preset repetition times and the ordinate is the RMSE in FIG. 12B. The three curves in the figure are results obtained when different values are adopted for the epoch respectively. It can be seen from FIG. 12B that, when epoch=5, epoch=10 or epoch=15, the RMSE decreases along with increase of the preset count repeat until getting close to 0 and then is kept unchanged. It can be seen from FIG. 12A and FIG. 12B that performing pixel diffusion for the preset repetition times may remarkably reduce the RMSE of the completed depth image, namely performing pixel extension for the preset repetition times may further improve the accuracy of the completed depth image.”)
Therefore, in combination of Wang, Smolic, Zhang and Wang_828, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a three-dimensional model for the source object based on the plurality of images of Smolic and training a first model to a training condition of Wang_828 with changing preset repetition times as seen in XU because this modification would be set preset repetition times according to a practical requirement ([0221] of XU).
Thus, the combination of Wang, Smolic, Zhang, Wang_828 and XU teaches wherein the preset training condition comprises: iteratively training the first model for a first preset number of times based on the sparse point cloud of the target object; and reducing a learning rate of the first model in response to iterative training being performed for a second preset number of times, wherein the second preset number of times is less than the first preset number of times.
Regarding claim 13, Wang, Smolic, Zhang and Wang_828 teach the electronic device according to claim 12, Remaining limitations of claim 13 is similar scope to claim 3 and therefore rejected under the same rationale.
4. Claims 4-5 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Mallet et al., U.S Patent No.9747716 (“Mallet”)
Regarding claim 4, Wang, Smolic, Zhang teach the method according to claim 1, wherein generating a plurality of animation models for the target object based on the first video comprises:
sampling various video frames of the first video to obtain a plurality of key video frames (see at least [0021] of Wang “The digital video generation system can extract poses in a variety of different forms. For example, in one or more embodiments, the digital video generation system generates a pose signature that disentangles the pose and appearance of an actor. For example, the digital video generation system generates the pose signature by combining different representations or poses of the actor (e.g., a DensePose representation and an OpenPose representation) for an input frame of the motion source. For instance, the digital video generation system generates the pose signature for the input frame by combining digital poses in the form of dense body UV-dimensional maps and predicted keypoint images with skeleton, face, and hand landmarks. [0037] of Zhang “By way of example, and not limitation, to generate the tracking mesh sequence, the system 102 may construct a double-layered deformation graph using both Linear Blend Skinning (LBS/skinning weights) and an on-body node graph (also referred to as a nonrigid deformation graph) from the constructed 3D model associated with the selected key-frame. For the selected key-frame of the input human-dynamics sequence, initial values of parameters of articulate motion, nonrigid motion and the deformation of each node on the double-layered deformation graph may be used as an initial guess to solve an objective function (i.e. a hybrid optimization function) to estimate the final values of the parameters of the articulate motion, the nonrigid motion and the deformation of each node on the double-layered deformation graph. In at least one embodiment, the system 102 may also include albedo and/or lighting information in the tracking sequence up to a number of frames of the input human-dynamics sequence.”; [0052] At 302D, key-frame selection may be performed. For key-frame selection, the circuitry 202 may select, from the input human-dynamics sequence, a key-frame for which the estimated pose is closest to a reference human pose, for example, a T-pose. In at least one embodiment, the circuitry 202 may compare the estimated pose of the human object 116 in each frame of the input human-dynamics sequence with a reference human pose based on a threshold distance-measure. Thereafter, based on the comparison, the circuitry 202 may estimate a key-frame score for each frame of the input human-dynamics sequence and select, from the input human-dynamics sequence, a key-frame for which the estimated key-frame score is a maximum”) );
generating one animation model for the target object using a second model based on one key video frame of the plurality of key video frames (see at least [0023] of Wang “Additionally, in some embodiments, the digital video generation system generates a motion embedding based on multiple digital poses in the sequence of digital poses. For example, the digital video generation system generates a motion signature that includes a representation of the movement that occurs from frame-to-frame as captured in multiple poses in the sequence of digital poses. In particular embodiments, the digital video generation system determines the motion signature by sampling poses from the motion source according to an uneven sampling distribution. For instance, in some embodiments, the digital video generation system utilizes an imbalanced sampling distribution of poses weighted closer in time to the input pose (e.g., for improved motion representation). In particular embodiments, the motion signature comprises a plurality of UV-dimensional map and predicted keypoint image combinations. The digital video generation system then uses a motion embedding model (e.g., another convolutional neural network) to extract motion features from the motion signature and generate a motion embedding”).;
extracting deformation information of the target object from each of the plurality of key video frames (see at least [0096] of Wang “ Similarly, the digital video generation system 106 utilizes a motion embedding model 412 to generate a motion embedding 416 based on the motion signature 408. For example, the motion embedding model 412 extracts motion features from the motion signature 408 to capture dynamic appearance changes for a variety of motion-dependent shape and appearance details— including loose garment deformations like wrinkles, folds, and flare. In particular, the motion embedding model 412 encodes the motion signature 408 into one-dimensional motion features (e.g., with dimension 2048) represented by the motion embedding 416.”0; and
adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame to obtain the animation model of the target object in each key video frame (see at least [0083] of Wang “ In one or more embodiments, the digital video generation system 106 uses the loss 328 to update or modify one or more learned parameters of the character animation neural network 208. In some embodiments, the digital video generation system 106 applies the loss 328 to each of the pose embedding model 306, the motion embedding model 310, the refinement embedding model 314, and the generative neural network 318. For example, the digital video generation system 106 minimizes function (3) with respect to the pose embedding model 306, the motion embedding model 310, and the refinement embedding model 314. Moreover, the digital video generation system 106 can apply the L.sub.GAN loss (e.g., maximize the loss) with respect to the discriminator model 320. In this way, the digital video generation system 106 trains the character animation neural network 208 in an end-to-end manner. In other embodiments, the digital video generation system 106 applies the loss 328 in a portion-wise manner to the character animation neural network 208 (e.g., such that different portions of the loss 328 correspond to different models) ; [0096] Similarly, the digital video generation system 106 utilizes a motion embedding model 412 to generate a motion embedding 416 based on the motion signature 408. For example, the motion embedding model 412 extracts motion features from the motion signature 408 to capture dynamic appearance changes for a variety of motion-dependent shape and appearance details— including loose garment deformations like wrinkles, folds, and flare. In particular, the motion embedding model 412 encodes the motion signature 408 into one-dimensional motion features (e.g., with dimension 2048) represented by the motion embedding 416. [0123] “As mentioned above, the digital video generation system 106 can retarget a source motion once the digital video generation system 106 trains the character animation neural network 208 for a specific character. FIG. 12 illustrates experimental results of implementing the digital video generation system 106 to perform motion retargeting in accordance with one or more embodiments. In particular, FIG. 12 shows additional examples of the digital video generation system 106 synthesizing plausible garment deformations of loose garments under complex motion sequences while also maintaining high quality visual results. Moreover, FIG. 12 shows that the digital video generation system 106 can flexibly train the character animation neural network 208 on different target characters to perform motion retargeting.”; [0142] As another example of an additional act not shown in FIG. 14, act(s) in the series of acts 1400 may include an act of: adjusting convolutional weights of the generative neural network according to the motion embedding; and generating, using the adjusted convolutional weights of the generative neural network, the frame of the digital video from the refined pose-motion embedding” [0053] of Zhang “ At 302E, deformation graph generation may be performed. The circuitry 202 may generate a double-layered deformation graph for the selected key-frame based on the constructed 3D human model. The double-layered deformation graph may include Linear Blend skinning (LBS) parameters for the articulate motion and an on-body node graph (i.e. a non-rigid deformation graph) for the non-rigid motion. Additionally, the double-layered deformation graph may include rigid deformation parameters for a rigid motion of a set of 3D points on a 3D human mesh of the human object 116.”; [0057] In at least one embodiment, the circuitry 202 may establish a correspondence (e.g., a per-vertex correspondence) for albedo parameters and lighting parameters of the 3D human mesh from the selected key-frame and up to the selected number of frames. The correspondence may be established based on the objective function that may also include a light regularization term and an albedo term. In such a case, the tracking sequence may be generated further based on the established correspondence for the albedo parameters and the lighting parameters of the 3D human mesh of the human object 116.”; ) In addition, the same motivation is used as the rejection for claim 1. Wang, Smolic, Zhang are understood to be silent on the remaining limitations of claim 4.
In the same field of endeavor, Mallet teaches wherein extracting deformation information of the target object; adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame to obtain the animation model of the target object in each key video frame (see at least col.5, lines 42-67-col.6, lines 1-34 “The animation model produced from these contours along with other information may be realized in a variety of forms. For example, the model may implement a collection of deformable geometries that correspond to shapes of the actor's face (e.g., shapes patterned after human facial muscles). Such shapes (referred to as blendshapes) have geometries that may be adjusted (e.g., weighted) so the model to is able to represent a particular facial expression (included in the actor's performance) from a range of expressions. While such blendshapes can provide an editable animation that substantially matches the actor's performance, linear combinations of the blendshapes often cannot accurately reproduce complex nonlinear interactions of the actor's skin. To address the situation, the computer system 102 takes corrective action by producing one or more additional shapes (referred to as corrective shapes) that are applied to the base blendshapes for accurately representing the captured facial expressions. One or more techniques or methodologies may be implemented to produce such corrective shapes. For example, using spatial wavelet basis functions, a pseudo-inverse solver technique may be implemented to produce wavelet basis functions that provide corrective shapes. In some situations, corrective shapes are often needed to address two or more potentially conflicting base blendshapes. While interfering blendshapes may provide a basis for introducing one or more corrective shapes, such corrective shapes may also be introduced due to other sources that cause presented facial expressions not to match the captured expressions. Referring to FIG. 2, in some arrangements the animation system 100 produces the animation model 108 (both shown in FIG. 1) from input information that includes the captured imagery (e.g., frames from video streams, etc.) of the actor's facial performance along with a deformable mesh of vertices that represents a base expression of the actor's face (e.g., a neutral facial expression) and a set of blendshapes for changing the facial expression being represented. From this information, the animation system 100 produces the animation model 108 for reconstructing the facial performance on the deformable mesh (such as a rig) to match the features of the captured imagery.”; col.9,lines 15-54)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic and Zhang with editing deformable geometries of model as seen in Mallet because this modification would represent a particular facial expression included in the actor's performance ( col.5, lines 42-55 of Millet)
Thus, the combination of Wang, Smolic, Zhang and Mallet teaches wherein generating a plurality of animation models for the target object based on the first video comprises: sampling various video frames of the first video to obtain a plurality of key video frames; generating one animation model for the target object using a second model based on one key video frame of the plurality of key video frames; extracting deformation information of the target object from each of the plurality of key video frames; and adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame to obtain the animation model of the target object in each key video frame.
Regarding claim 5, Wang, Smolic, Zhang and Mallet teach the method according to claim 4, wherein extracting deformation information of the target object from each of the plurality of key video frames comprises:
extracting posture information and shape information of the target object from each of the plurality of key video frames (see at least [0070] of Wang “ As used herein a pose embedding model refers to a model that can be tuned (e.g., trained) based on inputs to generate embeddings from poses. As part of a character animation neural network, a pose embedding model can include a variety of different machine learning models and/or neural networks trained to extract and encode spatial pose features based on a pose signature”;; [0090] “To illustrate, the digital video generation system 106 can use one or more different approaches to generating digital poses of an actor depicted in an image frame of the motion source. In some embodiments, the digital video generation system 106 generates a digital pose by determining a shape, outline, segmentation, or structural approximation of an actor depicted in an image frame of the motion source. In other embodiments, the digital video generation system 106 generates a digital pose by performing object reconstruction (e.g., using depth maps) to reconstruct a three-dimensional surface of the actor based on sampled surface depth points. Still, in other embodiments, the digital video generation system 106 generates a digital pose by generating a three-dimensional representation of the actor using a skinned multi-person linear model.” [0091] In certain embodiments, the digital video generation system 106 generates a digital pose by generating a dense body mapping image that includes a DensePose representation of a character portrayed in the image frame. In the dense body mapping image, the digital video generation system 106 represents each body part of a depicted actor in an image-space UV coordinate map. Additionally or alternatively, the digital video generation system 106 generates a digital pose by generating a keypoint data image that includes an OpenPose representation of a character portrayed in the image frame. In the keypoint data image, the digital video generation system 106 represents an actor's keypoints or anatomical landmarks such as skeleton, face, and hand positioning. In one or more embodiments, the dense body mapping image and/or the keypoint data image include three-channel images (e.g., RGB images) of the same size as the image frame from the motion source.”; [0095] Subsequently, the digital video generation system 106 utilizes a pose embedding model 410 to generate a pose embedding 414 based on the pose signature 406. In particular embodiments, the pose embedding model 410 extracts spatial pose features from the pose signature 406 for later conditioning a generative neural network 424 to synthesize an image of a character in motion. Specifically, the pose embedding model 410 encodes the pose signature 406 into spatial pose features represented by the pose embedding 414”; [0049] of Zhang “For instance, the neural network 104 may extract features associated with the human joints 308A . . . 308N of the human object 116 and the face landmarks 310A . . . 310N from the sequence of frames 306A . . . 306N of the RGB video. The human joints 308A . . . 308N and the face landmarks 310A . . . 310N may be associated with parameters of articulate motion of the human object 116. By way of example, and not limitation, the neural network 104 may be a “state of the art network”, as described in, Cao et al, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, arXiv:1611.08050v2, which is incorporated herein for reference. In such a case, the neural network 104 may include multiple stages, such as a first stage of N (e.g., 10) layers of the VGGNet to create feature maps for each input, i.e. each frame of the sequence of frames 306A . . . 306N. In other stages, a branched multi-stage Convolutional Neural Network (CNN) may be implemented on the feature maps. The first branch may output a set of 2D confidence maps of body parts at respective locations (for example, elbow, knee, shoulder, etc.). The second branch may output a set of 2D vector fields of part affinities, which may encode an extent of association between part affinities. A greedy inference model may be implemented on the set of 2D confidence maps and the set of 2D vector fields to estimate features associated with the human joints 308A . . . 308N and the face landmarks 310A . . . 310N of the human object 116 in each frame of RGB video.”; 050] At 302C, 3D human prior model generation may be performed. For 3D human prior model generation, the circuitry 202 may construct a 3D human model (i.e. a human parametric model) for each frame in the sequence of frames 306A . . . 306N of the RGB video by fitting a template human model on the estimated pose and the received geometry information 304A . . . 304N. Herein, a sequence of 3D human models 312A . . . 312N may be obtained by fitting the template human model on the estimated pose in each frame of the RGB video and the received geometry information 304A . . . 304N. The template human model may be a 3D template which may fit all body shapes, deform naturally to fit any human body pose, and even allow for soft tissue level or muscle level deformations.”; col.5, lines 42-67-col.1, lines 1-3 of Mallet “The animation model produced from these contours along with other information may be realized in a variety of forms. For example, the model may implement a collection of deformable geometries that correspond to shapes of the actor's face (e.g., shapes patterned after human facial muscles). Such shapes (referred to as blendshapes) have geometries that may be adjusted (e.g., weighted) so the model to is able to represent a particular facial expression (included in the actor's performance) from a range of expressions. While such blendshapes can provide an editable animation that substantially matches the actor's performance, linear combinations of the blendshapes often cannot accurately reproduce complex nonlinear interactions of the actor's skin. To address the situation, the computer system 102 takes corrective action by producing one or more additional shapes (referred to as corrective shapes) that are applied to the base blendshapes for accurately representing the captured facial expressions. One or more techniques or methodologies may be implemented to produce such corrective shapes. For example, using spatial wavelet basis functions, a pseudo-inverse solver technique may be implemented to produce wavelet basis functions that provide corrective shapes. In some situations, corrective shapes are often needed to address two or more potentially conflicting base blendshapes. While interfering blendshapes may provide a basis for introducing one or more corrective shapes, such corrective shapes may also be introduced due to other sources that cause presented facial expressions not to match the captured expressions.); and adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame comprises: for each key video frame, adjusting the one animation model for the target object using the posture information and shape information of the target object extracted from the key video frame to obtain the animation model of the target object in each key video frame (see at least [0090] of Wang “To illustrate, the digital video generation system 106 can use one or more different approaches to generating digital poses of an actor depicted in an image frame of the motion source. In some embodiments, the digital video generation system 106 generates a digital pose by determining a shape, outline, segmentation, or structural approximation of an actor depicted in an image frame of the motion source. In other embodiments, the digital video generation system 106 generates a digital pose by performing object reconstruction (e.g., using depth maps) to reconstruct a three-dimensional surface of the actor based on sampled surface depth points. Still, in other embodiments, the digital video generation system 106 generates a digital pose by generating a three-dimensional representation of the actor using a skinned multi-person linear model.” [0127] To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source).; [0142] As another example of an additional act not shown in FIG. 14, act(s) in the series of acts 1400 may include an act of: adjusting convolutional weights of the generative neural network according to the motion embedding; and generating, using the adjusted convolutional weights of the generative neural network, the frame of the digital video from the refined pose-motion embedding.”; [0068] of Zhang “ E.sub.shading and E.sub.Ireg may be energies of shading term and lighting regularization term respectively. E.sub.shading may indicate that the shade of each deformed vertex under the estimated lighting of this frame may be consistent with corresponding color intensity of the target position found in E.sub.data. E.sub.Ireg may indicate that the estimated lighting of each frame remains constant along the input human-dynamics sequence. To be specific, the objective function indicates, for each frame of the input human-dynamics sequence, the shading for all vertices and the lighting of the scene may be consistent. For example, if a vertex position changes then the shade of the vertex position may also change in accordance with the position of the vertex.”;”; col.9, lines 15-55 of Mallet “Referring to FIG. 4, along with providing closely matching representations to an actor face, the produced animation models provide for other functionality such as editability. As illustrated in the figure, three images 400, 402, 404 are presented of an actor's rig that tracks the performed facial expression of an actor. Image 400 presents an initial facial expression of the actor in which both eyes are in open positions along with the actor's mouth. By allowing blendshapes associated with the animation model to be adjusted, the presented facial expression may be edited. For example, as shown in image 402, adjustments may be executed for placing the eye lids in a closed position and slightly raisin the upper lip. In image 404, other adjustments may be added such as raising both eyebrows to allow and animator, editor, etc. to produce a slightly different emotion. Once edited, the adjustments may be used for updating the animation model such that the newly adjusted facial expressions may be reconstructed by applying the appropriate weight or weights to the blendshapes associated with the animation model.) In addition, the same motivation used as the rejection for claim 4.
Regarding claim 14, Wang, Smolic, Zhang teach the electronic device according to claim 11, Remaining limitations of claim 14 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 15, Wang, Smolic, Zhang and Mallet teach the electronic device according to claim 14, Remaining limitations of claim 15 is similar scope to claim 5 and therefore rejected under the same rationale.
5. Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Mallet et al., U.S Patent No.9747716 (“Mallet”) further in view of JU, U.S Patent Application Publication No.20230196678 (“JU”)
Regarding claim 6, Wang, Smolic, Zhang and Mallet teach the method according to claim 4, wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object comprises:
determining a typical animation model from the plurality of animation models for the target object, wherein the typical animation model for the target object has a posture and shape consistent with the three-dimensional model for the source object (see at least [0124] of Wang “As shown, FIG. 12 includes digital poses 1202 and 1204. In a first experimental test, the digital video generation system 106 generates image frames 1206, 1208 based on the digital poses 1202, 1204 (and others not shown for corresponding motion signatures). In the first experimental test, the digital video generation system 106 uses the character animation neural network 208 trained on a first character shown in image frame 1200a (depicting the first character in a rest pose). The digital video generation system 106 then retargets the character to perform a dance sequence including the digital poses 1202, 1204 as illustrated in the image frames 1206, 1208.”; [0055] of Zhang “At 302F, tracking sequence generation may be performed. The circuitry 202 may generate a tracking sequence for a 3D human mesh, from the selected key-frame and up to a number of frames of the input human-dynamics sequence. The generated tracking sequence may include final values of parameters of articulate motion and non-rigid motion of a set of 3D points on the 3D human mesh of the human object 116. The tracking sequence may be associated different features, such as with body motion, cloth motion, muscle deformation, and facial expressions associated with the human object 116 in the input human-dynamics sequence. [0090] To illustrate, the digital video generation system 106 can use one or more different approaches to generating digital poses of an actor depicted in an image frame of the motion source. In some embodiments, the digital video generation system 106 generates a digital pose by determining a shape, outline, segmentation, or structural approximation of an actor depicted in an image frame of the motion source. In other embodiments, the digital video generation system 106 generates a digital pose by performing object reconstruction (e.g., using depth maps) to reconstruct a three-dimensional surface of the actor based on sampled surface depth points. Still, in other embodiments, the digital video generation system 106 generates a digital pose by generating a three-dimensional representation of the actor using a skinned multi-person linear model.”; col.9, lines 16-54 of Mallet);
aligning the three-dimensional model with the typical animation model to obtain an aligned three-dimensional model (see at least [0127] of Wang “To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source).;
transferring a plurality of skinning weights of the animation models for the target object to the aligned three-dimensional model (see at least [0037] of Zhang “ By way of example, and not limitation, to generate the tracking mesh sequence, the system 102 may construct a double-layered deformation graph using both Linear Blend Skinning (LBS/skinning weights) and an on-body node graph (also referred to as a nonrigid deformation graph) from the constructed 3D model associated with the selected key-frame. For the selected key-frame of the input human-dynamics sequence, initial values of parameters of articulate motion, nonrigid motion and the deformation of each node on the double-layered deformation graph may be used as an initial guess to solve an objective function (i.e. a hybrid optimization function) to estimate the final values of the parameters of the articulate motion, the nonrigid motion and the deformation of each node on the double-layered deformation graph. In at least one embodiment, the system 102 may also include albedo and/or lighting information in the tracking sequence up to a number of frames of the input human-dynamics sequence.”; [” 0072] FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502. [0073] For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown). The circuitry 202 may apply the generated tracking sequence for the first 3D human mesh on the textured 3D human mesh 506 of the second human object 504. Once applied, an FVV (e.g., an FVV frame 508) may be rendered to re-enact the performance of the first human object 502 by the second human object 504.”); and
generating a second video for the source object using the aligned three-dimensional model based on the plurality of skinning weights (see at least [0090] of Wang “To illustrate, the digital video generation system 106 can use one or more different approaches to generating digital poses of an actor depicted in an image frame of the motion source. In some embodiments, the digital video generation system 106 generates a digital pose by determining a shape, outline, segmentation, or structural approximation of an actor depicted in an image frame of the motion source. In other embodiments, the digital video generation system 106 generates a digital pose by performing object reconstruction (e.g., using depth maps) to reconstruct a three-dimensional surface of the actor based on sampled surface depth points. Still, in other embodiments, the digital video generation system 106 generates a digital pose by generating a three-dimensional representation of the actor using a skinned multi-person linear model.”; [0053] of Zhang “At 302E, deformation graph generation may be performed. The circuitry 202 may generate a double-layered deformation graph for the selected key-frame based on the constructed 3D human model. The double-layered deformation graph may include Linear Blend skinning (LBS) parameters for the articulate motion and an on-body node graph (i.e. a non-rigid deformation graph) for the non-rigid motion. Additionally, the double-layered deformation graph may include rigid deformation parameters for a rigid motion of a set of 3D points on a 3D human mesh of the human object 116.[” 0072] FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502. [0073] For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown). The circuitry 202 may apply the generated tracking sequence for the first 3D human mesh on the textured 3D human mesh 506 of the second human object 504. Once applied, an FVV (e.g., an FVV frame 508) may be rendered to re-enact the performance of the first human object 502 by the second human object 504.”;) In addition, the same motivation is used as the rejection for claim 4. Wang, Smolic, Zhang and Mallet are understood to be silent on the remaining limitations of claim 6.
In the same field of endeavor, JU teaches aligning the three-dimensional model with the typical animation model to obtain an aligned three-dimensional model (see at least [0049] In operation 310, the transformation apparatus may align a template model 411 with an input model 413 through global optimization of respectively matching body parts of the template model 411 illustrated in drawing 410 of FIG. 4A to body parts of the input model 413 such that a position, length, and angle of each body part of the template model 411 correspond to those of each body part of the input model 413. For example, the transformation apparatus may perform matching such that a position, length, angle, and pose of each body part of the template model 411 correspond to those of each body part of the input model 413. The transformation apparatus may adjust a size and a position of a second polygon included in a second mesh of the template model 411 such that an outer shape of the template model 411 matches closely to an outer shape of the input model 413. For example, the transformation apparatus may perform optimization in a vertical direction (key direction) through a process of increasing or decreasing a height of the template model 411, as illustrated in drawing 410. After the template model 411 and the input model 413 have the same height through optimization in a vertical direction, the transformation apparatus may perform optimization in a horizontal direction through a process of widening or narrowing a waist of the template model 411.”;[0064] The transformation apparatus may transplant the skeleton 735 of the template model 730 aligned with the input model 710 into a model skeleton of the input model 710. For example, the transformation apparatus may generate the model skeleton of the input model 710, based on rigging information corresponding to the skeleton 735 of the template model 730 aligned with the input model 710. The transformation apparatus may generate the skeleton of the input model 710 by reflecting the rigging information corresponding to the skeleton 735 of the aligned template model 730 to a corresponding position of the input model 710.”);
transferring a plurality of skinning weights of the animation models for the target object to the aligned three-dimensional model (see at least [0065] After the model skeleton of the input model 710 is generated, the transformation apparatus may generate, based on correlation information, skinning information (“first skinning information”) indicating a relationship between the skeleton of the input model 710 and a mesh of the input model 710. [0066] The skinning information may include a skinning weight indicating what percentage each of vertices of a mesh included in a specific model is attached to each bone of a skeleton of a corresponding model. The transformation apparatus may generate first skinning information by combining i) second skinning information indicating a connection relation between the skeleton 735 of the aligned template model 730 and a second mesh of the template model 730, and ii) the correlation information. Here, the “combining” may be construed as obtaining weights of the first skinning information based on (i) weights of the second skinning information and (ii) correlation between meshes of a template model and meshes of the aligned model.”);
the source object using the aligned three-dimensional model based on the plurality of skinning weights (see at least [0066] The skinning information may include a skinning weight indicating what percentage each of vertices of a mesh included in a specific model is attached to each bone of a skeleton of a corresponding model. The transformation apparatus may generate first skinning information by combining i) second skinning information indicating a connection relation between the skeleton 735 of the aligned template model 730 and a second mesh of the template model 730, and ii) the correlation information. Here, the “combining” may be construed as obtaining weights of the first skinning information based on (i) weights of the second skinning information and (ii) correlation between meshes of a template model and meshes of the aligned model.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic, Zhang and Mallet with aligning first model with second model as seen in JU because this medication would perform matching such that a pose of each body part of the fist model matches closer to that of each body part of the second model ([0050] of JU)
Thus, the combination of Wang, Smolic , Zhang , Mallet and JU teaches wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object comprises: determining a typical animation model from the plurality of animation models for the target object, wherein the typical animation model for the target object has a posture and shape consistent with the three-dimensional model for the source object; aligning the three-dimensional model with the typical animation model to obtain an aligned three-dimensional model; transferring a plurality of skinning weights of the animation models for the target object to the aligned three-dimensional model; and generating a second video for the source object using the aligned three-dimensional model based on the plurality of skinning weights.
Regarding claim 16, Wang, Smolic, Zhang, Mallet teach the electronic device according to claim 14, Remaining limitations of claim 16 is similar scope to claim 6 and therefore rejected under the same rationale.
6. Claims 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Mallet et al., U.S Patent No.9747716 (“Mallet”) further in view of JU, U.S Patent Application Publication No.20230196678 (“JU”) further in view of KADAM et al, U.S Patent Application Publication No.20240331204 (“KADAM”) further in view of NUMAKAMI et al, U.S Patent Application Publication No. 20240011790 (“NUMAKAMI”)
Regarding claim 7, Wang, Smolic , Zhang , Mallet and JU teach the method according to claim 6, wherein aligning the three-dimensional model with the typical animation model comprises:
rotating the three-dimensional model by a first angle based on a rotation vector to obtain a first three-dimensional model (see at least [0099] of JU “The processor 1130 may align the template model with the input model through global optimization of respectively matching body parts of the template model to body parts of the input model such that a position, length, and an angle of each body part of the template model correspond to those of each body part of the input model. The processor 1130 may align the template model with the input model through local optimization of finely adjusting the body parts of the template model respectively matched to the body parts of the input model.”);
determining a distance between the second three-dimensional model and the typical animation model; and determining the second three-dimensional model as an aligned three-dimensional model for the source object in response to the distance being less than a preset threshold ([0085] of Zhang “In accordance with an embodiment, the circuitry may be further configured to compare the estimated pose of the human object in each frame of the input human-dynamics sequence with a reference human pose based on a threshold distance-measure. The reference human pose may be, for example, a T-pose. The circuitry may be further configured to estimate a key-frame score for each frame of the input human-dynamics sequence based on the comparison. The circuitry may be further configured to select, from the input human-dynamics sequence, the key-frame for which the estimated key-frame score may be a maximum”; col.10, lines 30-67of Mallet “.Referring to FIG. 6(b), the shape of the actor's mouth is mapped to the much wider mouth of the creature by aligning the corner point 602 to a corner point of a corresponding contour of the creature's mouth. As illustrated, due to the elongated size of the creatures mouth, the shape of the actor's mouth does not map and the shape for of the creature's mouth has little if any resemblance to shape of the actor's mouth. To provide an improved mapping, one or more techniques may be implemented. For example, the corners of the creatures mouth may be adjusted my closing one or more portions of the creatures mouth such that the speaking portion of the mouth more closely resembles the actor's mouth. Referring to FIG. 6(c), the corner of the creature's mouth is adjusted such that the creature in effect appears to talk using a smaller portion of its mouth. In this example, the extreme right side portion of the creature's mouth is sealed to effectively move the corner of the creature's mouth closer towards its midpoint. One or more techniques may be implemented to define the portion of the creature's mouth to seal for adjusting the corner point. For example, a vertical separation threshold may be defined such that the creature's mouth is sealed at the points where the threshold is not met or exceeded. In one arrangement, the separation threshold is defined as a percentage (e.g., 50%) of the vertical separation distance (“D”) at the midpoint of the creature's mouth (as represented by the pair of corresponding points 610 and 612). For point pairs that do not meet or exceed this separation threshold, the respective portion of the character's mouth is sealed. For example, the vertical distance separation between point pair 604a and 604b, point pair 606a and 606b, and point pair 608a and 608b are each compared to the defined threshold (e.g., 50% of D).”;[0057] of JU “The transformation apparatus may generate a correlation between the resized model 550 and the template model 510. The transformation apparatus may generate barycentric coordinates 560 between second vertices of the second polygon included in the second mesh of the template model 510 and a polygon included in a mesh of the resized model 550. [0058] The transformation apparatus may reflect the updated size of the resized model 550 to the input model 530). In addition, the same motivation is used as the rejection for claim 6. Wang, Smolic, Zhang, Mallet and JU are understood to be silent on the remaining limitations of claim 7.
In the same field of endeavor, KADAM teaches rotating the three-dimensional model by a first angle based on a rotation vector to obtain a first three-dimensional model, wherein the first angle is an angle indicated by the rotation vector (see at least [0038] In one or more examples, a global rigid transformation consisting of 3D rotation, 3D scaling, and 3D translation that optimally aligns the predicted right mesh with the original right mesh is calculated. Then, the global transformation is applied to all the vertices of the predicted right mesh to generate the new predicted right mesh (e.g., “transformed predicted right mesh”). Further, a local displacement vector is attached to each vertex of the transformed predicted right mesh. In one or more examples, the global transformation can be calculated and applied to set of partially cut meshes, based on which way is more optimal. Applying the global transformation to partial meshes may be helpful in scenarios where meshes exhibit local rigidity. The steps to calculate global and local symmetric displacements are detailed below, in accordance with one or more embodiments.”);
displacing the first three-dimensional model along a first direction by a first distance based on a displacement vector to obtain a second three-dimensional model, wherein the first direction and the first distance are a direction and a distance indicated by the displacement vector respectively (see at least [0038] In one or more examples, a global rigid transformation consisting of 3D rotation, 3D scaling, and 3D translation that optimally aligns the predicted right mesh with the original right mesh is calculated. Then, the global transformation is applied to all the vertices of the predicted right mesh to generate the new predicted right mesh (e.g., “transformed predicted right mesh”). Further, a local displacement vector is attached to each vertex of the transformed predicted right mesh. In one or more examples, the global transformation can be calculated and applied to set of partially cut meshes, based on which way is more optimal. Applying the global transformation to partial meshes may be helpful in scenarios where meshes exhibit local rigidity. The steps to calculate global and local symmetric displacements are detailed below, in accordance with one or more embodiments”.[0048] FIGS. 5A, 5B, and 6 illustrates the proposed framework of the present disclosure. For simplicity, only the mesh vertices are shown. FIG. 4(A) illustrates the predicted right mesh (darker circles) and the original right mesh (lighter circles). FIG. 4(B) illustrates the transformed predicted right mesh (darker circles) and the original right mesh (lighter circles). FIG. 6 illustrates local displacement vectors for each vertex of the transformed predicted right mesh. Accordingly, as illustrated in FIGS. 5A, 5B, and 6, the predicted right mesh is aligned with the original right mesh using the global transformation. Subsequently, the local displacements are found.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic, Zhang ,Mallet, JU with attaching each vertex of the transformed mesh as seen in KADAM because this medication would indicate the difference between the predicted attributes (e.g., vertex coordinates) and the original attributes ([0003] of KADAM) Wang, Smolic, Zhang ,Mallet, JU, KADAM are understood to be silent on the remaining limitations of claim 7.
In the same field of endeavor, NUMAKAMI teaches determining a distance between the second three-dimensional model and the typical animation mode (see at least [0055] In step S403 (a degree of region similarity calculation step), the degree of region similarity calculating unit 122 calculates the degree of region similarity between the first region and the second region based on the first region information and the second region information. In the present embodiment, the degree of region similarity calculation unit 122 calculates the sum of the distance between corresponding points on model surfaces of a 3D model of the first CAD data and a 3D model of the second CAD data.”); determining the second three-dimensional model as an aligned three-dimensional model for the source object in response to the distance being less than a preset threshold (see at least [0056] In addition, calculation is performed such that the smaller the sum of the distance is, the higher the degree of shape similarity is, and the larger the sum of the distance, the lower the degree of shape similarity is. That is, step S403 calculates the degree of similarity between the first region and the second region based on the first region information relating to the first region and the second region information relating to the second information”; [0059] In addition, in a case in which the degree of shape similarity in the degree of region similarity that has been calculated is at or above a predetermined threshold, it is judged that it is possible to reuse the first SLAM map, and it is determined that it is possible to duplicate the first SLAM map that was created in the first region and to create this to serve as the second SLAM map for the second region. That is, in a case in which the degree of similarity is at or above the predetermined threshold, the first map information is used to serve as the second map information.” Where degree of shape similarity above a predetermined threshold is considered as distance smaller the sum of distance is)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic, Zhang, Mallet, JU, KADAM with calculating the sum of the distance between corresponding points on model surfaces of a 3D model as seen in NUMAKAMI because this medication would calculate the degree of shape similarity based on the sum of the distance between corresponding points on model surfaces of 3D model ([0072] of NUMAKAMI)
Thus, the combination of Wang, Smolic, Zhang ,Mallet, JU, KADAM, NUMAKAMI teaches wherein aligning the three-dimensional model with the typical animation model comprises: rotating the three-dimensional model by a first angle based on a rotation vector to obtain a first three-dimensional model, wherein the first angle is an angle indicated by the rotation vector; displacing the first three-dimensional model along a first direction by a first distance based on a displacement vector to obtain a second three-dimensional model, wherein the first direction and the first distance are a direction and a distance indicated by the displacement vector respectively; determining a distance between the second three-dimensional model and the typical animation model; and determining the second three-dimensional model as an aligned three-dimensional model for the source object in response to the distance being less than a preset threshold.
Regarding claim 8, Wang, Smolic, Zhang ,Mallet, JU, KADAM, NUMAKAMI teach the method according to claim 7, wherein determining a distance between the second three-dimensional model and the typical animation model comprises:
for each point in the second three-dimensional model, determining a distance between the point and a corresponding point in the typical animation model as a first distance of the point; and determining a sum of first distances of various points in the second three-dimensional model as the distance between the second three-dimensional model and the typical animation model (see at least [0085] of Zhang “In accordance with an embodiment, the circuitry may be further configured to compare the estimated pose of the human object in each frame of the input human-dynamics sequence with a reference human pose based on a threshold distance-measure. The reference human pose may be, for example, a T-pose. The circuitry may be further configured to estimate a key-frame score for each frame of the input human-dynamics sequence based on the comparison. The circuitry may be further configured to select, from the input human-dynamics sequence, the key-frame for which the estimated key-frame score may be a maximum”; col.10, lines 30-67of Mallet “.Referring to FIG. 6(b), the shape of the actor's mouth is mapped to the much wider mouth of the creature by aligning the corner point 602 to a corner point of a corresponding contour of the creature's mouth. As illustrated, due to the elongated size of the creatures mouth, the shape of the actor's mouth does not map and the shape for of the creature's mouth has little if any resemblance to shape of the actor's mouth. To provide an improved mapping, one or more techniques may be implemented. For example, the corners of the creatures mouth may be adjusted my closing one or more portions of the creatures mouth such that the speaking portion of the mouth more closely resembles the actor's mouth. Referring to FIG. 6(c), the corner of the creature's mouth is adjusted such that the creature in effect appears to talk using a smaller portion of its mouth. In this example, the extreme right side portion of the creature's mouth is sealed to effectively move the corner of the creature's mouth closer towards its midpoint. One or more techniques may be implemented to define the portion of the creature's mouth to seal for adjusting the corner point. For example, a vertical separation threshold may be defined such that the creature's mouth is sealed at the points where the threshold is not met or exceeded. In one arrangement, the separation threshold is defined as a percentage (e.g., 50%) of the vertical separation distance (“D”) at the midpoint of the creature's mouth (as represented by the pair of corresponding points 610 and 612). For point pairs that do not meet or exceed this separation threshold, the respective portion of the character's mouth is sealed. For example, the vertical distance separation between point pair 604a and 604b, point pair 606a and 606b, and point pair 608a and 608b are each compared to the defined threshold (e.g., 50% of D).”;[0057] of JU “The transformation apparatus may generate a correlation between the resized model 550 and the template model 510. The transformation apparatus may generate barycentric coordinates 560 between second vertices of the second polygon included in the second mesh of the template model 510 and a polygon included in a mesh of the resized model 550. [0058] The transformation apparatus may reflect the updated size of the resized model 550 to the input model 530 [0055] of NUMAKAMI “In step S403 (a degree of region similarity calculation step), the degree of region similarity calculating unit 122 calculates the degree of region similarity between the first region and the second region based on the first region information and the second region information. In the present embodiment, the degree of region similarity calculation unit 122 calculates the sum of the distance between corresponding points on model surfaces of a 3D model of the first CAD data and a 3D model of the second CAD data. [0056] In addition, calculation is performed such that the smaller the sum of the distance is, the higher the degree of shape similarity is, and the larger the sum of the distance, the lower the degree of shape similarity is. That is, step S403 calculates the degree of similarity between the first region and the second region based on the first region information relating to the first region and the second region information relating to the second information.”) In addition, the same motivation is used as the rejection for claim 7.
Regarding claim 17, Wang, Smolic, Zhang ,Mallet, JU, teach the electronic device according to claim 16, Remaining limitations of claim 17 is similar scope to claim 7 and therefore rejected under the same rationale.
Regarding claim 18, Wang, Smolic, Zhang ,Mallet, JU, KADAM, NUMAKAMI teach the electronic device according to claim 17, Remaining limitations of claim 18 is similar scope to claim 8 and therefore rejected under the same rationale.
7. Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Mallet et al., U.S Patent No.9747716 (“Mallet”) further in view of JU, U.S Patent Application Publication No.20230196678 (“JU”) further in view of Kopp et al, U.S Patent Application Publication No.20240185518 (“Kopp”)
Regarding claim 9, Wang, Smolic, Zhang ,Mallet, JU teach the method according to claim 6, wherein generating a second video for the source object by using the aligned three-dimensional model based on the plurality of skinning weights comprises:
controlling the aligned three-dimensional model based on the plurality of skinning weights to obtain a three-dimensional animation for the source object (see at least [0037] of Zhang “By way of example, and not limitation, to generate the tracking mesh sequence, the system 102 may construct a double-layered deformation graph using both Linear Blend Skinning (LBS/skinning weights) and an on-body node graph (also referred to as a nonrigid deformation graph) from the constructed 3D model associated with the selected key-frame. For the selected key-frame of the input human-dynamics sequence, initial values of parameters of articulate motion, nonrigid motion and the deformation of each node on the double-layered deformation graph may be used as an initial guess to solve an objective function (i.e. a hybrid optimization function) to estimate the final values of the parameters of the articulate motion, the nonrigid motion and the deformation of each node on the double-layered deformation graph. In at least one embodiment, the system 102 may also include albedo and/or lighting information in the tracking sequence up to a number of frames of the input human-dynamics sequence.”; [0052] At 302D, key-frame selection may be performed. For key-frame selection, the circuitry 202 may select, from the input human-dynamics sequence, a key-frame for which the estimated pose is closest to a reference human pose, for example, a T-pose. In at least one embodiment, the circuitry 202 may compare the estimated pose of the human object 116 in each frame of the input human-dynamics sequence with a reference human pose based on a threshold distance-measure. Thereafter, based on the comparison, the circuitry 202 may estimate a key-frame score for each frame of the input human-dynamics sequence and select, from the input human-dynamics sequence, a key-frame for which the estimated key-frame score is a maximum” ; [0032] of JU “Skinning” refers to the process of assigning meshes with weights on bones of a skeleton to reflect the movement of the bones to the meshes. The skinning information may indicate relationships between bones of a skeleton of a corresponding model (for example, a template model) and meshes of the corresponding model. For example, the skiing information indicates how much weight is given transformation of bones in a skeleton to deform each of vertices of the mesh included in the corresponding model attached to the skeleton.”; [0066] The skinning information may include a skinning weight indicating what percentage each of vertices of a mesh included in a specific model is attached to each bone of a skeleton of a corresponding model. The transformation apparatus may generate first skinning information by combining i) second skinning information indicating a connection relation between the skeleton 735 of the aligned template model 730 and a second mesh of the template model 730, and ii) the correlation information. Here, the “combining” may be construed as obtaining weights of the first skinning information based on (i) weights of the second skinning information and (ii) correlation between meshes of a template model and meshes of the aligned model. [0067] The transformation apparatus may transform the input model 710, based on the correlation information and the skinning information. Depending on an example embodiment, when the input model 710 includes the rigging information, the transformation apparatus may change a pose of the input model using the rigging information of the input model.”); and
determining a two-dimensional projection of the three-dimensional animation for the source object from a preset perspective as the second video ( see at least col.7, lines 50-63 of Mallet “In one arrangement, a correspondence may be defined by projecting the vertices of the silhouette contour 308 and aligning end-points of the contour with end-points of the selected curve 306. Rather using a single curve, in some arrangements, multiple curves may be used for tracking a mouth area. For example, two curve segments may be used for tracking a lower lip and two segments may be used for tracking an upper lip. In one example, each segment may be defined from the middle of the respective lip to a corresponding end point of the lip. Once the points are defined, a mapping technique (e.g., an arc-length based mapping) may be used to define correspondences between the curves and the silhouette contour defined by the inner lips of the actor.”) In addition, the same motivation is used as the rejection for claim 6. Wang, Smolic, Zhang ,Mallet, JU are understood to be silent on the remaining limitations of claim 9.
In the same field endeavor, Kopp teaches determining a two-dimensional projection of the three-dimensional animation for the source object from a preset perspective as the second video (see at least [0276] FIG. 7A illustrates fitting of a 3D model of a dental arch to an image of a face, in accordance with an embodiment of the present disclosure. A position and orientation for the 3D model is determined relative to cropped frame 701. The 3D model at the determined position and orientation is then projected onto a 2D surface (e.g., a 2D plane) corresponding to the plane of the frame. Cropped frame 316 is fit to the 3D model, where dots 702 are vertices of the 3D model projected onto the 2D image space [0384] At block 1260, processing logic determines a plane to project the second 3D model onto based on a result of the fitting. Processing logic then projects the second 3D model onto the determined plane, resulting in a sketch in 2D showing the contours of the teeth from the second 3D model (e.g., the estimated future condition of the teeth from the same camera perspective as in the frame). A 3D virtual model showing the estimated future condition of a dental arch may be oriented such that the mapping of the 3D virtual model into the 2D plane results in a simulated 2D sketch of the teeth and gingiva from a same perspective from which the frame was taken. [0453] At block 1860, processing logic determines a plane to project the second 3D model onto based on a result of the fitting. Processing logic then projects the second 3D model onto the determined plane, resulting in a sketch in 2D showing the contours of the objects in the area of interest from the second 3D model (e.g., the estimated future condition of the area of interest from the same camera perspective as in the frame). A 3D virtual model showing the estimated future condition of area of interest may be oriented such that the mapping of the 3D virtual model into the 2D plane results in a simulated 2D sketch of the area of interest from a same perspective from which the frame was taken”; )
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic, Zhang ,Mallet and aligning first model with second model as seen in JU with projecting 3D model to determined plane as seen in Kopp because this medication would fit of a 3D model to an image ([0276] of Kopp)
Thus, the combination of Wang, Smolic, Zhang ,Mallet, JU and Kopp teaches wherein generating a second video for the source object by using the aligned three-dimensional model based on the plurality of skinning weights comprises: controlling the aligned three-dimensional model based on the plurality of skinning weights to obtain a three-dimensional animation for the source object; and determining a two-dimensional projection of the three-dimensional animation for the source object from a preset perspective as the second video.
Regarding claim 19, Wang, Smolic, Zhang ,Mallet, JU teach the electronic device according to claim 14, Remaining limitations of claim 19 is similar scope to claim 9 and therefore rejected under the same rationale.
8. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al, U.S Patent Application Publication No.20230123820 (“Wang”) in view of Smolic et al, U.S Patent Application Publication No.20200320727 (“Smolic”) further in view of Zhang, U.S Patent Application Publication No.20220130056 (Zhang) further in view of Mallet et al., U.S Patent No.9747716 (“Mallet”) further in view of JU, U.S Patent Application Publication No.20230196678 (“JU”) further in view of Kopp et al, U.S Patent Application Publication No.20240185518 (“Kopp”) further in view of Xu et al U.S Patent Application Publication No.20250175694 (“Xu”)
Regarding claim 10, Wang, Smolic, Zhang ,Mallet, JU and Kopp teach the method according to claim 9, further comprising: determining background information based on the first video, wherein the background information does not include the target object (see at least “[0127] of Wang “To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source).”; [0005] of Smolic “Current FVV techniques for generating FVV visual information from real-world content include image-based techniques, where the intermediate views between cameras are generated using interpolation or warping of the available images; and geometry-based techniques, where 3D geometry of both the dynamic foreground and static background is acquired, allowing the rendering from any other viewpoint.; [0072-0073] of Zhang;) and
embedding the background information of the first video into the two-dimensional projection as an optimized second video for the source object (see at least [0127] of Wang “To perform such motion retargeting, the digital video generation system 106 can perform certain acts to compensate for differences (e.g., different body proportions) between the actor from the motion source and the character trained on the character animation neural network 208. For example, in some embodiments, the digital video generation system 106 performs alignment modifications. Additionally or alternatively, the digital video generation system 106 adjusts the height and width of the detected skeletons. Similarly, in certain implementations, the digital video generation system 106 displaces the target character. For instance, the digital video generation system 106 moves the target character up or down within an image frame so that the target character appears to stand on the ground of a target background (or a background from the motion source); [0072] of Zhang “ FIG. 5 is a diagram that illustrates an exemplary scenario for 4-Dimensional (4D) video reenactment by the system of FIG. 2, in accordance with an embodiment of the disclosure. FIG. 5 is explained in conjunction with elements from FIGS. 1, 2, 3A, 3B, and 4. With reference to FIG. 5, there is shown a diagram 500 that illustrates an exemplary scenario for 4D video reenactment. In the diagram 500, there is shown a first human object 502 and a second human object 504. The first human object 502 may be an original performer (e.g., a freestyle soccer player, as shown) of an act/performance which may be scanned via the scanning setup 108 in the 3D environment 114. Based on operations described in FIGS. 3A, 3B, and 4, the circuitry 202 may generate a tracking sequence for a first 3D human mesh for the first human object 502. [0073] For the second human object 504 to reenact the act/performance of the first human object 502, the second human object 504 may be scanned via the scanning setup 108 and a textured 3D human mesh 506 of the second human object 504 may be generated in a reference pose, such as a T-pose (as shown). The circuitry 202 may apply the generated tracking sequence for the first 3D human mesh on the textured 3D human mesh 506 of the second human object 504. Once applied, an FVV (e.g., an FVV frame 508) may be rendered to re-enact the performance of the first human object 502 by the second human object 504.[ [0276] of Kopp “ FIG. 7A illustrates fitting of a 3D model of a dental arch to an image of a face, in accordance with an embodiment of the present disclosure. A position and orientation for the 3D model is determined relative to cropped frame 701. The 3D model at the determined position and orientation is then projected onto a 2D surface (e.g., a 2D plane) corresponding to the plane of the frame. Cropped frame 316 is fit to the 3D model, where dots 702 are vertices of the 3D model projected onto the 2D image space”) In addition, the same motivation is used as the rejection for claim 9. Wang, Smolic, Zhang ,Mallet, JU and Kopp are understood to be silent on the remaining limitations of claim 10.
In the same field of endeavor, Xu teaches determining background information based on the first video, wherein the background information does not include the target object ; embedding the background information of the first video as an optimized second video for the source object (see at least [0048] Deep learning models may contain a person as one of the semantic categories, the result will tell whether there is a person segment large enough to be considered as a foreground. If not, the input image will go through another deep learning model [M2] to get an embedding of the image, represented as a vector. If there is a foreground person segment, this semantic foreground part will be cut out from the background, and the remaining background will be in-painted with a deep learning model [M3], before going through the deep learning model [M2] to get background embeddings. The foreground image is person specific and will go through another deep learning model [M4] to extract person-specific embeddings, this model could be obtained by fine-tuning the model [M2] to person attribute-related recognition tasks, like gender, age, and identity. [0153] Likewise at block 660, the background embedding, which involves extracting scenery or background-related attributes, is performed. The background embedding details are then sent to the server to find reference images that include similar background-related attributes.[0178] The process of blocks 1300 and 1350 may be applied to query images as a whole image or query images that may be segmented into foreground and background to obtain the best images from the image dataset 1310. For example, if the query image includes a person (or an object of interest) in the foreground, then the query image may be segmented into a foreground containing the person (object of interest) and a background without the person (object of interest). As such, the control circuitry may segment the background from the foreground, as described in relation to blocks 610-630 of FIG. 6 and generate an embedding vector 1365 for the background and another embedding vector 1365 for the foreground. The control circuitry may then concatenate the background embedding vector together with the foreground person embedding vector. The concatenation, in some embodiments, may take relevant portions of the vector and combine them such that they make logical sense. For example, in a portrait of a person with a scenic background, the concatenation may include relevant portions of the foreground and the background in a combined vector. This combined vector may then be used as an inquiry vector and inputted into to generate the vector index 1370 as described earlier.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of generating a digital video through person-specific appearance modeling and motion retargeting of Wang, Smolic, Zhang ,Mallet, JU, and projecting 3D model to determined plane of Kopp with embedding the background as seen in Xu because this medication would find reference images that include similar background-related attributes ([0153] of Xu)
Thus, the combination of Wang, Smolic, Zhang ,Mallet, JU, Kopp and Xu teaches further comprising: determining background information based on the first video, wherein the background information does not include the target object; and embedding the background information of the first video into the two-dimensional projection as an optimized second video for the source object.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614