DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Preliminary Amendment
The preliminary amendment received 03/12/2025 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-9, 15-18, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Kim (03-12-2025-IDS; NPL; Deep Video Portraits) and in view of Watson (US 20170318283 A1).
Regarding to claim 1, Kim discloses a method (page 162:1; left column: enable photo-realistic re-animation of portrait videos using only an input video; page 162:2; left column; 1 Introduction: synthesize a photo-realistic video portrait of a target actor that mimics the actions of a source actor; enable a source actor to take full control of the rigid head pose, face expressions and eye motion of the target actor; page 163:3; Fig. 2: deep video portraits enable a source actor to fully control a target video portrait) comprising:
receiving semantic description data representative of a three-dimensional (3D) model of a face of a user (page 163:1; left column: receive and transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor; page 163:3; Fig. 2: receive source parameters, i.e. pose, expression, and eyes, from a source video; obtain a low-dimensional parametric representation (left) of both videos using monocular face reconstruction; the pose, expression, and eyes of source video are semantic description data;
PNG
media_image1.png
376
992
media_image1.png
Greyscale
; page 163:5; right column: we stack all Nw conditioning inputs of a time window in a 3D tensor X of size W × H × 9Nw);
determining a head pose of the face of the user in an interactive video (page 163:1; left column: determine and transfer the full 3D head position and head rotation; the head position and head rotation include head pose; page 163:3; Fig. 2: target parameters, i.e. identity, pose, are derived from a target video as illustrated in Fig. 2;
PNG
media_image2.png
140
312
media_image2.png
Greyscale
; page 163:4; 3 Overview; left column: Full target video frames are synthesized; interactive parameter-based video control; page 163:6; right column; 7.1 Applications: all parameter dimensions are estimated and transferred from a source video sequence or edited manually through an interactive user interface);
determining a parametric model of a lighting of an environment of the interactive video (page 163:3; Fig. 2: the illumination, i.e. lighting, of the target video is obtained with the target parameters;
PNG
media_image2.png
140
312
media_image2.png
Greyscale
; page 163:4; left column; 3 Overview: the resulting sequence of low dimensional parameter vectors represents the actor’s identity, head pose, expression, eye gaze, and the scene lighting for every video frame; page 163:5; right column; 6 Rendering-to-video-translation: consistent lighting effects between foreground and background.);
synthesizing the face of the user using the semantic description data with the head pose and under the parametric model of the lighting of the environment of the interactive video (page 163:1; left column: receive and transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor; page 163:3; Fig. 2: the pose, expression and eyes characteristics of the source actor are applied to the target video, which keeping the illumination and identify of the target actor;
PNG
media_image3.png
246
392
media_image3.png
Greyscale
;
PNG
media_image4.png
190
172
media_image4.png
Greyscale
; page 163:3; left column; 3 Overview: transfer the head pose, expression, and/or eye gaze parameters from the source to the target; generate new synthetic renderings of the target actor based on the modified parameters); and
generating a modified interactive video comprising an image of the synthesized face of the user in the interactive video (page 163:3; Fig. 2: at the end of the rendering to video translation network, the synthesized face is rendered in the environment of the target actor as illustrated in Fig. 2;
PNG
media_image5.png
356
976
media_image5.png
Greyscale
; page 163:4; left column; 3 Overview: convert the synthetic input into photo-realistic output; several video rewrite applications).
Kim fails to explicitly disclose interactive video is immersive video.
In same field of endeavor, Watson teaches interactive video is immersive video (Fig. 1; [0009]: immersive interactive video system; Fig. 2; [0018]: create an immersive interactive video; Fig. 5; [0022]: an exemplary method of creating an immersive interactive video system; [0023]: create a 3D model of the scene of the video content based on the plurality of frames of the video content; [0024]: the 360° display allows viewer 405 to be fully immersed in the scene displayed on display 495).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kim to include interactive video is immersive video as taught by Watson. The motivation for doing so would have been to adjust the scene being displayed on display 195 based on the position of the viewer in the room relative to display; to create an immersive interactive video; to record up to 360° field of view; to allow viewer 405 to be fully immersed in the scene displayed on display; to construct a synthetic view of the scene and to allow the viewer to view the scene from the view point of the dynamic virtual camera as it moves through the scene as taught by Watson in Fig. 2, and paragraphs [0013], [0018], [0024], and [0025].
Regarding to claim 2, Kim in view of Watson discloses the method of claim 1, wherein semantic description data representative of a 3D model of a face of a user (same as rejected in claim 1) comprises:
an indication of an identity representative of a physiognomy of the user with a neutral expression (Kim; page 163:3; Fig. 2: identities with a neutral expression as illustrated in Fig. 2;
PNG
media_image6.png
250
414
media_image6.png
Greyscale
; page 163:4; left column; 3 Overview: the target’s identity and appearance; the actor’s identity; page 163:7: Fig. 5: neutral expressions:
PNG
media_image7.png
212
150
media_image7.png
Greyscale
);
an indication of an expression representative of an emotional expression or a deformation of the face incurred by uttering speech with respect to the neutral expression of the user (or is optional; Kim; page 163:4; left column; 3 Overview: facial expression, and eye motion of a source actor; page 163:7; Fig. 5; Fig. 6: mouths move left as illustrated in Fig. 5;
PNG
media_image8.png
226
230
media_image8.png
Greyscale
); and
an indication of an appearance representative of a reflectance of the face of the user (Kim; page 163:2; right column: represents face geometry, reflectance, and motion as well as eye gaze and eye blinks; page 163:3; Fig. 2: face colors of face are indicated in Fig. 2;
PNG
media_image9.png
108
126
media_image9.png
Greyscale
; page 163:3; right column: re-rendering the target model with photorealistic reflectance, i.e. indication; page 163:4; left column; 3 Overview: a normal color rendering; page 163:5; left column: a color rendering; page 163:5; right column: the modified target actor model under the estimated target illumination, while keeping the target identity; page 163:7; Fig. 5; Fig. 6: the indication of face color is shown in Fig. 5 and Fig. 6).
Regarding to claim 3, Kim in view of Watson discloses the method of claim 2 wherein the indication of an identity is a 3D mesh representing a 3D geometry of the face with a neutral expression (Kim; page 163:3; left column; Fig. 2: a template mesh of face;
PNG
media_image10.png
118
62
media_image10.png
Greyscale
; page 163:4; right column: model geometry variation through an affine model v∈R3N that stacks per-vertex deformations of the underlying template mesh with N vertices; page 163:5; right column: texture the head model with a constant unique gradient texture map).
Regarding to claim 4, Kim in view of Watson discloses the method of claim 3 wherein the indication of expression comprises a plurality of displacements of vertices of the 3D mesh incurred by the emotional expression (Kim; page 163:3; left column; Fig. 2: non-rigidly deforming a template mesh; page 163:4; right column: model geometry variation through an affine model v∈R3N that stacks per-vertex deformations, i.e. displacements, of the underlying template mesh with N vertices).
Regarding to claim 7, Kim in view of Watson discloses the method of claim 1 wherein the generating of the modified immersive video further comprises:
generating an image of the synthesized face of the user against a uniform background (Kim; page 163:3; Fig. 2: the pose, expression and eyes characteristics of the source actor are applied to the target video, which keeping the illumination and identify of the target actor;
PNG
media_image3.png
246
392
media_image3.png
Greyscale
;
PNG
media_image4.png
190
172
media_image4.png
Greyscale
; page 163:3; left column; 3 Overview: transfer the head pose, expression, and/or eye gaze parameters from the source to the target; generate new synthetic renderings of the target actor based on the modified parameters; page 163:5; right column; 6 Rendering-to-video translation: synthesize the appropriately modified and filled-in background, including even some consistent lighting effects between foreground and background; page 163:7; Fig. 5: the shadow in the background of the second row moves consistently with the modified foreground head motion); and
compositing the generated image of the synthesized face of the user extracted from the uniform background into the interactive video (Kim; page 163:3; Fig. 2: at the end of the rendering to video translation network, the synthesized face is rendered in the environment of the target actor as illustrated in Fig. 2;
PNG
media_image5.png
356
976
media_image5.png
Greyscale
; page 163:4; left column; 3 Overview: convert the synthetic input into photo-realistic output; several video rewrite applications; page 163:5; right column; 6 Rendering-to-video translation: synthesize the appropriately modified and filled-in background, including even some consistent lighting effects between foreground and background).
Kim in view of Watson further discloses immersive video (Watson; Fig. 1; [0009]: immersive interactive video system; Fig. 2; [0018]: create an immersive interactive video; Fig. 5; [0022]: an exemplary method of creating an immersive interactive video system; [0023]: create a 3D model of the scene of the video content based on the plurality of frames of the video content; [0024]: the 360° display allows viewer 405 to be fully immersed in the scene displayed on display 495).
Same motivation of claim 1 is applied here.
Regarding to claim 8, Kim in view of Watson discloses the method of claim 1 wherein the generating takes at input the synthesized face of the user and the modified interactive video at a previous frame (Kim; page 163:4; left column; 3 Overview: The resulting sequence of low dimensional parameter vectors represents the actor’s identity, head pose, expression, eye gaze, and the scene lighting for every video frame; page 163:4; right column; 3 Overview: monocular face tracker reconstructs Np =261 parameters per video frame; page 163:5; left column: the identity is only estimated in the first frame and is kept constant afterwards; all other parameters are estimated every frame; page 163:5; left column: 5 Synthetic Conditioning Input: all parameters are copied in a relative manner from the source to the target, i.e., with respect to a neutral reference frame.).
Kim in view of Watson further discloses immersive video (Watson; Fig. 1; [0009]: immersive interactive video system; Fig. 2; [0018]: create an immersive interactive video; Fig. 5; [0022]: an exemplary method of creating an immersive interactive video system; [0023]: create a 3D model of the scene of the video content based on the plurality of frames of the video content; [0024]: the 360° display allows viewer 405 to be fully immersed in the scene displayed on display 495).
Same motivation of claim 1 is applied here.
Regarding to claim 9, Kim in view of Watson discloses the method of claim 8 wherein the generating of the modified immersive video (same as rejected in claim 1 and claim 8) uses a generative adversarial network (Kim; page 163:2; right column: our approach is a conditional generative adversarial network (cGAN), which is specifically tailored to video portrait synthesis; page 163:4; left column: propose a generative model to synthesize people in clothing; page 163:5; right column; 6. Rendering-to-video translation: conditional generative adversarial network; page 163:6; right column: we train our networks using the Tensor- Flow deep learning framework).
Regarding to claim 15, Kim discloses an apparatus, comprising one or more processors configured to (page 162:1; left column: enable photo-realistic re-animation of portrait videos using only an input video; page 162:2; left column; 1 Introduction: synthesize a photo-realistic video portrait of a target actor that mimics the actions of a source actor; enable a source actor to take full control of the rigid head pose, face expressions and eye motion of the target actor; page 163:3; Fig. 2: deep video portraits enable a source actor to fully control a target video portrait; GPU renderer; page 163:5; left column: the CPU):
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 15.
Regarding to claim 16, Kim in view of Watson discloses the apparatus of claim 15 wherein semantic description data representative of a 3D model of a face of a user (same as rejected in claim 1) comprises:
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 16.
Regarding to claim 17, Kim in view of Watson discloses the apparatus of claim 16,
The rest claim limitations are similar to claim limitations recited in claim 3. Therefore, same rational used to reject claim 3 is also used to reject claim 17.
Regarding to claim 18, Kim in view of Watson discloses the apparatus of claim 17,
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 18.
Regarding to claim 21, Kim in view of Watson discloses the apparatus of claim 15,
wherein to generate the modified immersive video, one or more processors are further configured to (same as rejected in claim 15):
The rest claim limitations are similar to claim limitations recited in claim 7. Therefore, same rational used to reject claim 7 is also used to reject claim 21.
Regarding to claim 22, Kim in view of Watson discloses the apparatus of claim 15,
wherein to generate the modified immersive video, one or more processors (same as rejected in claim 15)
The rest claim limitations are similar to claim limitations recited in claim 8. Therefore, same rational used to reject claim 8 is also used to reject claim 22.
Regarding to claim 23, Kim in view of Watson discloses the apparatus of claim 22,
The rest claim limitations are similar to claim limitations recited in claim 9. Therefore, same rational used to reject claim 9 is also used to reject claim 23.
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kim (03-12-2025-IDS; NPL; Deep Video Portraits) in view of Watson (US 20170318283 A1), and further in view of Oz (US 20210358193 A1).
Regarding to claim 5, Kim in view of Watson discloses the method of claim 3,
Kim in view of Watson fails to explicitly disclose:
wherein an indication of appearance comprises a reflectance on a surface of the 3D mesh.
In same field of endeavor, Oz teaches wherein an indication of appearance comprises a reflectance on a surface of the 3D mesh ([0139]: a texture map is a 2D image in which each color pixel represents the red, green and blue reflectance coefficients of a certain area in the 3D model; [0207]: a reflectance map details the color of each part of the face or body; [0234]: the color values of the model are related to polygons in a 3D mesh or to pixels in a 2D texture map; [0320]: a correspondence is found between the Hue and Saturation values of people in adequate exposures and the respective Brightness values of their skin; Fig. 12; [0371]: the triangular mesh template is pre-annotated with a predefined annotation of face parts; use UV coordinates for the mesh along with a 2D texture map for coloring face parts in different colors as in FIG. 12;
PNG
media_image11.png
182
572
media_image11.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kim in view of Watson to include wherein an indication of appearance comprises a reflectance on a surface of the 3D mesh as taught by Oz. The motivation for doing so would have been to provide an improved visual interaction between users that may emulate the visual interaction; to represent the red, green and blue reflectance coefficients of a certain area in the 3D model; to improve the natural appearance of the rendered model; to color face parts in different colors as taught by Oz in paragraphs [0131], [0139], [0146], and [0371].
Regarding to claim 19, Kim in view of Watson discloses the apparatus of claim 17,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 19.
Claims 6 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim (03-12-2025-IDS; NPL; Deep Video Portraits) in view of Watson (US 20170318283 A1), in view of Oz (US 20210358193 A1), and further in view of Tauber (US 20200195904 A1).
Regarding to claim 6, Kim in view of Watson discloses the method of claim 1 wherein the head pose of the face of the user in an immersive video (same as rejected in claim 1) comprises
Kim in view of Watson fails to explicitly disclose an indication of 3D rotation and translation of the face in an image of the immersive video with respect to a fronto-parallel viewpoint.
In same field of endeavor, Oz teaches an indication of 3D rotation and translation of the face in an image of the immersive video with respect to a viewpoint ([0062]: the updated representation of virtual 3D video conference environment represents the updated 3D participant representation information for at least some of the multiple participants; [0134]: the avatar of the fifth participant faces the avatar of first participant; [0136]: tracking the user's head pose and eye gaze may also be used to control the virtual world's appearance on the user's screen; if the user looks at the right side of the screen, the point of view of the virtual camera may move to the right; [0234]: several images of the model with some translation or rotation between them are used in order to create a grid at a higher resolution; [0379]: optimize and find a rotation and translation that optimally transforms the first image points to the second image points in the least-squares sense; [0575]: obtain participant interaction parameters during the 3D video conference).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kim in view of Watson to include an indication of 3D rotation and translation of the face in an image of the immersive video with respect to a viewpoint as taught by Oz. The motivation for doing so would have been to provide an improved visual interaction between users that may emulate the visual interaction; to represent the red, green and blue reflectance coefficients of a certain area in the 3D model; to improve the natural appearance of the rendered model; to color face parts in different colors as taught by Oz in paragraphs [0131], [0139], [0146], and [0371].
Kim in view of Watson and Oz fails to explicitly disclose a viewpoint is a fronto-parallel viewpoint.
In same field of endeavor, Tauber teaches a viewpoint is a fronto-parallel viewpoint ([0023]: plane sweeping occurs across depth deviations from the static scene in a single fronto-parallel direction; fronto-parallel is a direction that is parallel with the image plane; [0024]: a three-dimensional immersive video; in a 3D immersive video, end users can view the action of the event and move through the captured volume freely; [0029]: planes are selectively located in a fronto-parallel direction within the space with respect to a given camera view).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kim in view of Watson and Oz to include a viewpoint is a fronto-parallel viewpoint as taught by Tauber. The motivation for doing so would have been to enable a reduction in computational power used during the creation of a 3D reconstruction of a scene in real time; to produce high quality views with a minimum latency so that the visual result is immersive for the viewer; to reconstruct the 3D structure from a projection of the 3D structure on a series of images; to view the action of the event and move through the captured volume freely by end users in a 3D immersive video as taught by Tauber in paragraphs [0012], [0014-0015], and [0024].
Regarding to claim 20, Kim in view of Watson discloses the apparatus of claim 15.
The rest claim limitations are similar to claim limitations recited in claim 6. Therefore, same rational used to reject claim 6 is also used to reject claim 20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights