DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 1/15/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 112
1 The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
2 Claim 47 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
3 Claim 47 recites “…wherein at each iteration the relative camera pose is learned for the corresponding one of the plurality of pairs of sequential frames and the relative camera pose…” without any mention of a group of iterations within the claim as well as the parent claim 43. It is worth mentioning that claim 47 has similar limitations as of claim 22, which the parent claim (which is another dependent claim instead of an independent claim) of claim 22 does have mentions of a group of iterations ([Claim 21] reciting “The method of claim 1, wherein the global primitive-based representation of the video is progressively built over a plurality of iterations each associated with a corresponding one of the plurality of pairs of sequential frames.”). There is antecedent language in this claim, making the claim to be rejected under 35 U.S.C. 112(b), and is assumed a similar rejection as of claim 22 as mentioned below.
Claim Rejections - 35 USC § 103
4 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
5 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6 Claim(s) 1-4, 6, 13-18, 24, 28-30, 36-37, 41, 43-44, and 46 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1).
7 Regarding claim 1, Rochette teaches a method, comprising: at a device: learning relative camera poses for a plurality of pairs of sequential frames in a video of a static scene using a local primitive-based representation of a frame in each the pairs of sequential frames ([Page 1; Section I] reciting “We present a new three-step approach for novel view synthesis and motion transfer (see Fig. 1). We first infer the pose and appearance information of a subject from an image. The pose is then transferred to a novel view along with the appearance, from which we render diffuse primitives, using a realistic camera model, onto a high-dimensional image of the foreground of the scene…Disentangling shape and appearance of an image is fundamental for general 3D understanding and lies at the heart of our idea. If we focus on synthesizing novel views of humans, the localization of human joints in the three-dimensional space can be seen as a first step towards human body shape estimation.”; [Page 4; Section III (C)] reciting “From the input image I∗1 , we only estimate information about the foreground subject. As it is impossible to accurately infer a novel view of a static background captured by a static camera, we infer a constant background around the subject, and use a segmentation mask to discard the background information from the groundtruth image”);
performing view synthesis using the of the video ([Page 1; Section I] reciting “We present a new three-step approach for novel view synthesis and motion transfer (see Fig. 1). We first infer the pose and appearance information of a subject from an image. The pose is then transferred to a novel view along with the appearance, from which we render diffuse primitives, using a realistic camera model, onto a high-dimensional image of the foreground of the scene.”; [Page 6; Section IV (B)] reciting “As such, this work represents an initial step towards full-actor synthesis from arbitrary input videos.”).
8 Rochette does not explicitly teach progressively building a global primitive-based representation of the video, using the relative camera poses; and performing view synthesis using the global primitive-based representation of the video.
9 Taguchi teaches progressively building a global primitive-based representation of the video, using the relative camera poses; and performing view synthesis using the global primitive-based representation of the video ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”).
10 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette) to incorporate the teachings of Taguchi to provide a method for a type of global primitive representation that can teach the similar methods taught by the local representations and frames from Rochette. Doing so would allow the correspondence search and registration to be faster due to the smaller number of plane primitives, and can be performed in real time as stated by Taguchi ([0023] recited).
11 Regarding claim 2, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the local primitive-based representation of the frame in each the pairs of sequential frames is a two-dimensional (2D) Gaussian representation (Rochette; [Page 2; Section I] reciting “The key insight that allows this to take place is the use of diffuse Gaussian primitives to move from sparse 3D joint locations into a dense 2D feature image via a novel density renderer”).
12 Regarding claim 3, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the local primitive-based representation of the frame in each the pairs of sequential frames is a three-dimensional (3D) Gaussian representation (Rochette; [Page 4; Section III (B)] reciting “We render a simplified skeletal structure of diffuse primitives, directly obtained from the 3D pose. The intuition underlying our new renderer is straightforward. Each primitive can be understood as an anisotropic Gaussian defined by its location μ and shape Σ, and the rendering operation is the process of integrating along each ray.”).
13 Regarding claim 4, Rochette in view of Taguchi teaches the method of claim 1, wherein the local primitive-based representation of the frame in each the pairs of sequential frames is parameterized by color, rotation, scale, and opacity (Rochette; [Page 3; Section III] reciting “The pose P1 is transferred to a new viewpoint using camera extrinsic parameters (R1→2,t1→2). From the pose P2, seen from a novel orientation, we derive the location µ2 and shape Σ2 of the primitives, which are used, along with their appearance a1 and the intrinsic parameters and distortion coefficients of the second camera (K2,D2), for the rendering of the subject in a high-dimensional image J2.”; [Page 4; Section III (B)] reciting “Here, fR calculates the rotation between two non-zero vectors (see Appendix II) and wij loosely represents the width of the limb.”; [Page 4; Section III (B)] reciting “Here, fR calculates the rotation between two non-zero vectors (see Appendix II) and wij loosely represents the width of the limb.”; [Page 4; Section III (B)] reciting “We derive the weights ωijk quantifying the influence of each primitive (μk,Σk) (including the background) onto each ray rij , such that ∀k ∈ [1..M + 1],”; [Page 4; Section III (B)] reciting “We define the rays rij as unit vectors originating from the pinhole, distorted by the lens, and passing through every pixel of the image,”).
14 Regarding claim 6, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the local primitive-based representation of the frame is learned ([Page 3, Section III] reciting “We jointly learn pose estimation as part of the view synthesis process, using our Gaussian-based renderer, as shown in Fig. 2. From the input image I∗ 1 , our model encodes two modalities, the three-dimensional pose P1 relative to input camera, and the appearance a1 of the subject.”).
15 Regarding claim 13, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein a relative camera pose is learned for every adjacent pair of frames in the video (Rochette; [Page 6; Section IV] reciting “We require pairs of images for training, therefore we have |E| = |F| × |V|2 possible pairs, with F and V, referring to the sets of available frames and views respectively. This gives us a total of 81.6M pairs of images for training, 11.7M pairs for validation and 12.7M for testing”).
16 Regarding claim 14, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein a relative camera pose is learned for a subset of all adjacent pairs of frames in the video (Rochette; [Page 6; Section IV] reciting “We require pairs of images for training, therefore we have |E| = |F| × |V|2 possible pairs, with F and V, referring to the sets of available frames and views respectively. This gives us a total of 81.6M pairs of images for training, 11.7M pairs for validation and 12.7M for testing”).
17 Regarding claim 15, Rochette in view of Taguchi teaches the method of claim 1, wherein the of the video is a two-dimensional (2D) Gaussian representation (Rochette; [Page 2; Section I] reciting “The key insight that allows this to take place is the use of diffuse Gaussian primitives to move from sparse 3D joint locations into a dense 2D feature image via a novel density renderer”).
18 Taguchi as mentioned previously in claim 1 can further teach the following limitations, specifically the global primitive-based representation ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”; [0034] reciting “The keypoints can be selected using 3D keypoint detectors from the 3D point cloud without using a texture image. Example 3D keypoint detectors include Normal Aligned Radial Feature (NARF) and 3D Speeded Up Robust Feature (SURF). Alternatively, the system can select 2D keypoints from each texture image using 2D keypoint detectors and back-project the keypoints using the corresponding depth value to obtain the 3D point primitives.”).
19 As explained in the rejection of claim 1, the obviousness for combining of the global primitive-based representation of Taguchi into Rochette’s local representations is provided above.
20 Regarding claim 16, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the of the video is a three-dimensional (3D) Gaussian representation (Rochette; [Page 4; Section III (B)] reciting “We render a simplified skeletal structure of diffuse primitives, directly obtained from the 3D pose. The intuition underlying our new renderer is straightforward. Each primitive can be understood as an anisotropic Gaussian defined by its location μ and shape Σ, and the rendering operation is the process of integrating along each ray.”).
21 Taguchi as mentioned previously in claim 1 can further teach the following limitations, specifically the global primitive-based representation ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”; [0034] reciting “The keypoints can be selected using 3D keypoint detectors from the 3D point cloud without using a texture image. Example 3D keypoint detectors include Normal Aligned Radial Feature (NARF) and 3D Speeded Up Robust Feature (SURF). Alternatively, the system can select 2D keypoints from each texture image using 2D keypoint detectors and back-project the keypoints using the corresponding depth value to obtain the 3D point primitives.”).
22 As explained in the rejection of claim 1, the obviousness for combining of the global primitive-based representation of Taguchi into Rochette’s local representations is provided above.
23 Regarding claim 17, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the of the video is parameterized by color, rotation, scale, and opacity (Rochette; [Page 3; Section III] reciting “The pose P1 is transferred to a new viewpoint using camera extrinsic parameters (R1→2,t1→2). From the pose P2, seen from a novel orientation, we derive the location µ2 and shape Σ2 of the primitives, which are used, along with their appearance a1 and the intrinsic parameters and distortion coefficients of the second camera (K2,D2), for the rendering of the subject in a high-dimensional image J2.”; [Page 4; Section III (B)] reciting “Here, fR calculates the rotation between two non-zero vectors (see Appendix II) and wij loosely represents the width of the limb.”; [Page 4; Section III (B)] reciting “Here, fR calculates the rotation between two non-zero vectors (see Appendix II) and wij loosely represents the width of the limb.”; [Page 4; Section III (B)] reciting “We derive the weights ωijk quantifying the influence of each primitive (μk,Σk) (including the background) onto each ray rij , such that ∀k ∈ [1..M + 1],”; [Page 4; Section III (B)] reciting “We define the rays rij as unit vectors originating from the pinhole, distorted by the lens, and passing through every pixel of the image,”).
24 Taguchi as mentioned previously in claim 1 can further teach the following limitations, specifically the global primitive-based representation ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”; [0034] reciting “The keypoints can be selected using 3D keypoint detectors from the 3D point cloud without using a texture image. Example 3D keypoint detectors include Normal Aligned Radial Feature (NARF) and 3D Speeded Up Robust Feature (SURF). Alternatively, the system can select 2D keypoints from each texture image using 2D keypoint detectors and back-project the keypoints using the corresponding depth value to obtain the 3D point primitives.”).
25 As explained in the rejection of claim 1, the obviousness for combining of the global primitive-based representation of Taguchi into Rochette’s local representations is provided above.
26 Regarding claim 18, Rochette in view of Taguchi teaches the method of claim 1, wherein the of the video is a model of the static scene of the video (Rochette; [Page 1; Section I] reciting “We present a new three-step approach for novel view synthesis and motion transfer (see Fig. 1). We first infer the pose and appearance information of a subject from an image. The pose is then transferred to a novel view along with the appearance, from which we render diffuse primitives, using a realistic camera model, onto a high-dimensional image of the foreground of the scene.”; [Page 4; Section III (C)] reciting “From the input image I∗1 , we only estimate information about the foreground subject. As it is impossible to accurately infer a novel view of a static background captured by a static camera, we infer a constant background around the subject, and use a segmentation mask to discard the background information from the groundtruth image”).
27 Taguchi as mentioned previously in claim 1 can further teach the following limitations, specifically the global primitive-based representation ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”).
28 As explained in the rejection of claim 1, the obviousness for combining of the global primitive-based representation of Taguchi into Rochette’s local representations is provided above.
29 Regarding claim 24, Rochette in view of Taguchi teaches the method of claim 1, wherein the view synthesis includes generating a novel view of the scene in the video (Rochette; [Abstract] reciting “We present a new approach for synthesizing novel views of people in new poses.”; [Page 6; Section IV (B) reciting “As such, this work represents an initial step towards full-actor synthesis from arbitrary input videos.”).
30 Regarding claim 28, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), wherein the view synthesis is performed for a 3D content creation application (Rochette; [Page 1; Section I] reciting “Disentangling shape and appearance of an image is fundamental for general 3D understanding and lies at the heart of our idea. If we focus on synthesizing novel views of humans, the localization of human joints in the three-dimensional space can be seen as a first step towards human body shape estimation.”).
31 Claims 29 and 43 has similar limitations as of claim 1, therefore they are rejected under the same rationale as claim 1.
32 Regarding claim 30, Rochette in view of Taguchi teaches the system of claim 29 (see claim 29 rejection above), wherein the local primitive-based representation of the frame in each the pairs of sequential frames is one of: a two-dimensional (2D) Gaussian representation, or a three-dimensional (3D) Gaussian representation (Rochette; [Page 2; Section I] reciting “The key insight that allows this to take place is the use of diffuse Gaussian primitives to move from sparse 3D joint locations into a dense 2D feature image via a novel density renderer”; [Page 4; Section III (B)] reciting “We render a simplified skeletal structure of diffuse primitives, directly obtained from the 3D pose. The intuition underlying our new renderer is straightforward. Each primitive can be understood as an anisotropic Gaussian defined by its location μ and shape Σ, and the rendering operation is the process of integrating along each ray.”).
33 Regarding claim 36, Rochette in view of Taguchi teaches the system of claim 29 (see claim 29 rejection above), wherein the of the video is one of: a two-dimensional (2D) Gaussian representation, or a three-dimensional (3D) Gaussian representation (Rochette; [Page 2; Section I] reciting “The key insight that allows this to take place is the use of diffuse Gaussian primitives to move from sparse 3D joint locations into a dense 2D feature image via a novel density renderer”; [Page 4; Section III (B)] reciting “We render a simplified skeletal structure of diffuse primitives, directly obtained from the 3D pose. The intuition underlying our new renderer is straightforward. Each primitive can be understood as an anisotropic Gaussian defined by its location μ and shape Σ, and the rendering operation is the process of integrating along each ray.”).
34 Taguchi as mentioned previously in claim 29 (and by extension claim 1) can further teach the following limitations, specifically the global primitive-based representation ([0023] reciting “In contrast to prior art methods that mainly use points for registration, the present method has the following advantages. The correspondence search and registration is faster due to the smaller number of plane primitives, and can be performed in real time. The method produces plane-based 3D models that are more compact than point-based models. The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0040] reciting “A pose 240 of the sensor i.e., a 6-DOF rigid body transformation with respect to a coordinate system of the global map) is determined by registering the primitives in the measurements with respect to the primitives in the landmarks in the global map. Our registration method using both points and planes and our RANSAC procedure are described below.”; [0034] reciting “The keypoints can be selected using 3D keypoint detectors from the 3D point cloud without using a texture image. Example 3D keypoint detectors include Normal Aligned Radial Feature (NARF) and 3D Speeded Up Robust Feature (SURF). Alternatively, the system can select 2D keypoints from each texture image using 2D keypoint detectors and back-project the keypoints using the corresponding depth value to obtain the 3D point primitives.”).
35 As explained in the rejection of claim 29 (and by extension as explained in the rejection of claim 1), the obviousness for combining of the global primitive-based representation of Taguchi into Rochette’s local representations is provided above.
36 Claim 37 has similar limitations as of claim 18, therefore it is rejected under the same rationale as claim 18.
37 Claim 41 has similar limitations as of claim 24, therefore it is rejected under the same rationale as claim 24.
38 Claim 44 has similar limitations as of claim 30, therefore it is rejected under the same rationale as claim 30.
39 Claim 46 has similar limitations as of claim 36, therefore it is rejected under the same rationale as claim 36.
40 Claim(s) 5, 19-20, 31, and 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claim 1 and 29, further in view of Jenkins et al. (US 6057847 A).
41 Regarding claim 5, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), but does not explicitly teach wherein the local primitive-based representation is of a first frame sequence-wise in the pair of sequential frames.
42 Jenkens teaches wherein the local primitive-based representation is of a first frame sequence-wise in the pair of sequential frames ([Page 97; Column 52, Lines 42-59] reciting “The present method includes a somewhat different dynamic load balancing strategy which tends to constrain each processor to process the same set of primitives in successive frames. This is achieved by resizing and repositioning subimage windows in a manner that causes them to substantially track the primitives in their own local display list. This approach produces a greater degree of temporal subimage coherence and results in less primitive redistribution. This would, in principle, cause primitives to remain within the subimage of their origin throughout their lifetime in the image stream. At first glance, this seems to be a somewhat Quixotic adventure given the seemingly heterogenous and unpredictable character of optical flow on the image plane. In general optical flow on the image plane is heterogenous and is heavily influenced by the depth structure of the environment. In fact, however, there is generally more coherence and uniformity in the image-plane flow field of a typical image sequence than is generally realized.”).
43 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Jenkins to provide a method that has a “sequence-wise” function for the sequential frames as well as the local representation that is taught by Rochette in view of Taguchi. Doing so would cause primitives to remain within the subimage of their origin throughout their lifetime in the image stream as stated by Jenkins ([Page 97; Column 52, Lines 50-52] recited).
44 Regarding claim 19, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), and although Taguchi could teach wherein the global primitive-based representation of the video is progressively built from an initialized global primitive-based representation of the video ([0023] reciting “The method provides global registration without suffering from local minima, or initialization problems as in the prior art local registration methods.”; [0066] reciting “Therefore, as shown in FIG. 3, we initialize the RANSAC procedure with a triplet of primitives, if available, in the following preferred order: 3 planes 301, 2 planes and 1 point 302, 1 plane and 2 points 303, or 3 points 304.”), prior art from Jenkins can teach this limitation further.
45 Jenkins teaches wherein the global primitive-based representation of the video is progressively built from an initialized global primitive-based representation of the video ([Page 79; Column 16, Lines 47-55] reciting “The resulting global visibility array reflects all visible primitives for a frame. This array is logically compared to the global visibility array for the previous frame to identify newly invisible primitives. While this method has a higher overhead in zeroing, initializing, and array comparison its cost is relatively independent of the number of newly occluded primitives and can be employed when temporal visibility coherence is very low.”).
46 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Jenkins to provide a clearer type of initialization for the building of the global representations, for which the global representations are provided from Rochette in view of Taguchi. Doing so would cause primitives to remain within the subimage of their origin throughout their lifetime in the image stream as stated by Jenkins ([Page 97; Column 52, Lines 50-52] recited).
47 Regarding claim 20, Rochette in view of Taguchi and Jenkins teaches the method of claim 19 (see claims 1 and 19 rejections above),
48 Jenkins from claim 19 can teach the limitations, specifically wherein the initialized global primitive-based representation of the video is generated with an orthogonal camera pose. ([Page 82; Column 22, Lines 14-18] reciting “Specifically, FIG. 15 shows a complete primitive reprojection cycle and starts at the transformation step 1500 in which each primitive on the local display list is transformed using the concatenated transformation matrix describing object and camera motion.”; [Page 86; Column 29, Lines 64-67; Column 30, Lines 1-2] reciting “The image-space extent of any primitive becomes vanishingly small as the primitive's normal becomes orthogonal to the view frustrum rays in its vicinity. For this reason alone ray-primitive intersections would seem not to be an efficient or reliable method of detecting newly visible primitives in an exposure region.”).
49 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi and Jenkins) to incorporate the teachings of Jenkins further to provide an orthogonal camera pose to go along with the creating of the initialized global representation from the teachings of Rochette in view of Taguchi and Jenkins. Doing so would cause primitives to remain within the subimage of their origin throughout their lifetime in the image stream as stated by Jenkins ([Page 97; Column 52, Lines 50-52] recited).
50 Claim 31 has similar limitations as of claim 5, therefore it is rejected under the same rationale as claim 5.
51 Regarding claim 38, Rochette in view of Taguchi teaches the system of claim 29 (see claim 29 rejection above), but does not explicitly teach wherein the global primitive-based representation of the video is progressively built from an initialized global primitive-based representation of the video, wherein the initialized global primitive-based representation of the video is generated with an orthogonal camera pose.
52 Jenkins teaches wherein the global primitive-based representation of the video is progressively built from an initialized global primitive-based representation of the video ([Page 79; Column 16, Lines 47-55] reciting “The resulting global visibility array reflects all visible primitives for a frame. This array is logically compared to the global visibility array for the previous frame to identify newly invisible primitives. While this method has a higher overhead in zeroing, initializing, and array comparison its cost is relatively independent of the number of newly occluded primitives and can be employed when temporal visibility coherence is very low.”), wherein the initialized global primitive-based representation of the video is generated with an orthogonal camera pose ([Page 82; Column 22, Lines 14-18] reciting “Specifically, FIG. 15 shows a complete primitive reprojection cycle and starts at the transformation step 1500 in which each primitive on the local display list is transformed using the concatenated transformation matrix describing object and camera motion.”; [Page 86; Column 29, Lines 64-67; Column 30, Lines 1-2] reciting “The image-space extent of any primitive becomes vanishingly small as the primitive's normal becomes orthogonal to the view frustrum rays in its vicinity. For this reason alone ray-primitive intersections would seem not to be an efficient or reliable method of detecting newly visible primitives in an exposure region.”).
53 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Jenkins to provide a type of initialization for the building of the global representations utilizing an orthogonal camera pose to go along with the desired methods, utilizing the global representations taught by the teachings of Rochette in view of Taguchi. Doing so would cause primitives to remain within the subimage of their origin throughout their lifetime in the image stream as stated by Jenkins ([Page 97; Column 52, Lines 50-52] recited).
54 Claim(s) 7-9 and 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claim 1 and 29, further in view of Guizilini et al. (US 20210398301 A1).
55 Regarding claim 7, Rochette in view of Taguchi teaches the method of claim 6, wherein the local primitive-based representation of the frame is learned by (see claims 1 and 6 rejections above):
generating a monocular depth for the frame (Rochette; [Page 7; Section IV (D)] reciting “However, using a monocular sequence of an unseen subject, we can quickly retrain both the appearance and image synthesis modules to the new individual. Our model is able to produce novel views of an unseen individual from a single camera.”),
56 Rochette in view of Taguchi does not explicitly teach generating an initialized local primitive-based representation of the frame with points lifted from the monocular depth, and beginning with the initialized local primitive-based representation, learning the local primitive-based representation of the frame by minimizing a loss between an image rendered from the local primitive-based representation and the frame.
57 Guizilini teaches generating an initialized local primitive-based representation of the frame with points lifted from the monocular depth, and beginning with the initialized local primitive-based representation, learning the local primitive-based representation of the frame by minimizing a loss between an image rendered from the local primitive-based representation and the frame ([Abstract] reciting “A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes training a monocular depth model and a monocular pose model to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network.”; [0021] reciting “In particular, existing monocular depth estimation methods rely on joint learning of depth and pose networks. The joint learning of depth and pose networks relies on a proxy photometric loss that enables the use of geometric cues as a single source of supervision.”; [0066] reciting “The simultaneous training includes minimizing the photometric re-projection error between the original target image I.sub.t 502 and the warped target image 552 Î.sub.t (e.g., synthesized images). An image synthesis operation of view synthesis block 550 is performed using, for example, spatial transformer networks (STNs), via grid sampling with bilinear interpolation.”).
58 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Guizilini to provide a method that is able to utilize the monocular depth by learning and minimizing a loss between specific images, utilizing the monocular depth and local representation methods provided by Rochette in view of Taguchi. Doing so would allow the models to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network as stated by Guizilini ([0005] recited).
59 Regarding claim 8, Rochette in view of Taguchi and Guizilini teaches the method of claim 7 (see claims 1 and 6-7 rejections above),
60 Guizilini from claim 7 can further teach the limitations, specifically wherein the monocular depth is generated using a monocular depth network ([Abstract] reciting “A method for monocular depth/pose estimation in a camera agnostic network is described. The method includes training a monocular depth model and a monocular pose model to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network.”).
61 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Guizilini to provide a type of network for the monocular depth that is proved by the teachings of Rochette in view of Taguchi. Doing so would allow the models to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network as stated by Guizilini ([0005] recited).
62 Regarding claim 9, Rochette in view of Taguchi and Guizilini teaches the method of claim 7 (see claims 1 and 6-7 rejections above), .
63 Guizilini from claim 7 can further teach the limitations, specifically wherein the loss is a photometric loss ([0021] reciting “In particular, existing monocular depth estimation methods rely on joint learning of depth and pose networks. The joint learning of depth and pose networks relies on a proxy photometric loss that enables the use of geometric cues as a single source of supervision.”).
64 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Guizilini to provide a photometric loss that can be part of the monocular depth methods that are taught by Rochette in view of Taguchi. Doing so would allow the models to learn monocular depth estimation and monocular pose estimation based on a target image and context images from monocular video captured by the camera agnostic network as stated by Guizilini ([0005] recited).
65 Claim 32 has similar limitations as of claim 7, therefore it is rejected under the same rationale as claim 7.
66 Claim(s) 10-11, 33-34, and 45 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claims 1, 29, and 43, further in view of Valentin et al. (US 20240037829 A1).
67 Regarding claim 10, Rochette in view of Taguchi teaches the method of claim 1, wherein the relative camera pose of each of the plurality of pairs of sequential frames is learned by (see claim 1 rejection above): but does not explicitly teach transforming the local primitive-based representation of the frame in the pair of sequential frames by a learnable affine transformation into the other frame in the pair of sequential frames.
68 Valentin teaches transforming the local primitive-based representation of the frame in the pair of sequential frames by a learnable affine transformation into the other frame in the pair of sequential frames ([Abstract] reciting “The method comprises receive a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model. For a pixel of the image the method computes a ray from a virtual camera through the pixel into the cage animated according to the animation data and computes a plurality of samples on the ray. Each sample is a 3D position and view direction in one of the 3D elements. The method computes a transformation of the samples into a canonical version of the cage to produce transformed samples. For each transformed sample, the method queries a learnt radiance field parameterization of the 3D scene to obtain a color value and an opacity value.”; [0051] reciting “In some examples the inputs 400 comprise default values for some or all of the deformation description, the viewpoint, the intrinsic camera parameters… The face or body tracker is a trained machine learning model which takes as input captured sensor data depicting at least part of a person's face or body and predicts values of parameters of a 3D face model or 3D body model of the person. The parameters are shape parameters, pose parameters or other parameters.”; [0065] reciting “Where the primitive 3D elements are spheres or cuboids the transform of the samples to the canonical cage is computed using affine transformations instead, which are expressive enough for large rigidly moving sections of the motion field.”).
69 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Valentin to provide a method that involves a learnable affine transformation for the types of frames as well as the local representation provided by Rochette in view of Taguchi. Doing so would allow these transformations to be expressive enough for large rigidly moving sections of the motion field as stated by Valentin ([0065] recited).
70 Regarding claim 11, Rochette in view of Taguchi and Valentin teaches the method of claim 10 (see claims 1 and 10 rejections above),
71 Valentin from claim 10 can further teach the limitations, specifically wherein the affine transformation is optimized by a loss between a rendered image of the frame when transformed by the affine transformation and the other frame in the pair of sequential frames ([0067] reciting “In the case where the 3D elements are tetrahedra, an optimization is optionally used to compute the transform at operation 406 by optimizing primitive point lookups. The optimization comprises computing the transformation P of a sample by setting P equal to a normalized distance between a previous and a next intersection of a tetrahedron on a ray”; [0103] reciting “Each sample is assigned an index of one of the 3D primitive elements of the cage according to the element the sample falls within. The samples are then transformed 806 to a canonical cage, which is a version of the cage in a rest position. The transformed samples are used to compute an output pixel color by using volume rendering. The output pixel color is compared with the ground truth output pixel color of the training image and the difference or error is assessed using a loss function. The loss function output is used to carry out backpropagation so as to train 808 the machine learning model and output a trained machine learning model 810.”).
72 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Valentin to provide a method that involves a learnable affine transformation that optimizes a loss between certain images/frames for the types of frames as well as the local representation provided by Rochette in view of Taguchi. Doing so would allow these transformations to be expressive enough for large rigidly moving sections of the motion field as stated by Valentin ([0065] recited).
73 Claims 33 and 45 has similar limitations as of claim 10, therefore they are rejected under the same rationale as claim 10.
74 Claim 34 has similar limitations as of claim 11, therefore it is rejected under the same rationale as claim 11.
75 Claim(s) 12 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) and Valentin et al. (US 20240037829 A1) as of claims 1, 10, 29, and 33, further in view of Brand et al. (US 20090148072 A1).
76 Regarding claim 12, Rochette in view of Taguchi and Valentin teaches the method of claim 10 (see claims 1 and 10 rejections above), but does not explicitly teach wherein during an optimization of the affine transformation, attributes of the local primitive-based representation of the frame are frozen.
77 Brand teaches wherein during an optimization of the affine transformation, attributes of the local primitive-based representation of the frame are frozen ([0036] reciting “For parameter estimation, the same system of linear equations can be solved for the optimal affine transforms {W.sub.l}.sub.l=1, . . . , |L 141, given paired training images, and marginal likelihoods of the labels for the source image X. This maximizes the lower bound, while holding the CRF parameters fixed. To estimate the CRF parameters while holding the transforms fixed, we optimize the lower bound”).
78 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi and Valentin) to incorporate the teachings of Brand to provide a method is able to “freeze” of have something from the affine transformation to be fixed, utilizing the affine transformation methods provided by Rochette in view of Taguchi and Valentin. Doing so would touch up a source image to produce a target image as stated by Brand ([Abstract] recited).
79 Claim 35 has similar limitations as of claim 12, therefore it is rejected under the same rationale as claim 12.
80 Claim(s) 21 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claims 1 and 29, further in view of Sideris et al. (US 20160110837 A1).
81 Regarding claim 21, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), but does not explicitly teach wherein the global primitive-based representation of the video is progressively built over a plurality of iterations each associated with a corresponding one of the plurality of pairs of sequential frames.
82 Sideris teaches wherein the global primitive-based representation of the video is progressively built over a plurality of iterations each associated with a corresponding one of the plurality of pairs of sequential frames ([0042] reciting “The tiler 21 then iterates through the predetermined subdivision of the frame into multiple tiles, in particular determining for each defined tile a primitive list of relevance to that tile (i.e. from the polygon list and vertex positions which it receives, it determines a primitive list per tile in the frame). Once this primitive list per tile has been determined, individual tiles are issued to the fragment frontend 31, under control of the job control unit 30, to be processed in the sequence of pipelined stages which follow. The fragment frontend 31 comprises the the iterator 22, the triangle setup unit 23, the rasterizer 24, the early z tester 25 and the fragment shader 26. Each tile is iteratively processed (selected by tile iterator 22) and within that tile a primitive is passed from the tile iterator 22 to the triangle setup unit 23 in order to determine the relevant triangle coefficients for rasterization.”).
83 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sideris to provide a method that has the global representation to be built off of iterations associated with various frames, utilizing the global representation methods and sequential frames taught by Rochette in view of Taguchi. Doing so would allow the methods of graphics processing to generate a frame of display data as stated by Sideris ([Abstract] recited).
84 Claim 39 has similar limitations as of claim 21, therefore it is rejected under the same rationale as claim 21.
85 Claim(s) 22-23 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) and Sideris et al. (US 20160110837 A1) as of claims 1, 29, and 39, further in view of Sucar et al. (US 20240005597 A1).
86 Regarding claim 22, Rochette in view of Taguchi and Sideris teaches the method of claim 21 (see claims 1 and 21 rejections above), and although Sideris could teach wherein at each iteration the relative camera pose is learned for the corresponding one of the plurality of pairs of sequential frames and the relative camera pose is used with the corresponding one of the plurality of pairs of sequential frames to update the global primitive-based representation of the video ([0024] reciting “In some embodiments the graphics processing apparatus is capable of accessing a frame buffer which is updated to hold the frame of display data by a frame buffer updating stage of the sequence of processing stages, and the frame buffer updating stage is capable of updating the frame buffer for the current tile in dependence on a comparison between a checksum value for the current tile and a stored checksum value for the corresponding tile of the previous frame at the same display position, and the graphics processing apparatus is capable of determining if the current tile has the predetermined value in dependence on the checksum value for the current tile.”; [0042] reciting “The tiler 21 then iterates through the predetermined subdivision of the frame into multiple tiles, in particular determining for each defined tile a primitive list of relevance to that tile (i.e. from the polygon list and vertex positions which it receives, it determines a primitive list per tile in the frame). Once this primitive list per tile has been determined, individual tiles are issued to the fragment frontend 31, under control of the job control unit 30, to be processed in the sequence of pipelined stages which follow. The fragment frontend 31 comprises the the iterator 22, the triangle setup unit 23, the rasterizer 24, the early z tester 25 and the fragment shader 26. Each tile is iteratively processed (selected by tile iterator 22) and within that tile a primitive is passed from the tile iterator 22 to the triangle setup unit 23 in order to determine the relevant triangle coefficients for rasterization.”), prior art from Sucar can teach this limitation further.
87 Sucar teaches wherein at each iteration the relative camera pose is learned for the corresponding one of the plurality of pairs of sequential frames and the relative camera pose is used with the corresponding one of the plurality of pairs of sequential frames to update the global primitive-based representation of the video ([0014] reciting “Selecting the set of pixels based on the loss probability distribution for example allows pixels to be sampled based on how useful they are likely to be in updating the model (e.g. how likely they are to correspond to parts of the environment with a large amount of detail and/or that are insufficiently represented by the model).”; [0045] reciting “Based on the loss, at least the camera pose estimate and the model are jointly optimised to generate an update to the camera pose estimate and an update to the model. This approach for example allows for a learning of the environment so as to iteratively improve the camera pose estimate and the model of the environment. Optimising the model in this manner for example improves the accuracy of the 3D representation of the environment generated using the model. This joint optimisation may be applied in a SLAM system in which, in parallel to the joint optimisation, a tracking system continuously optimises a camera pose estimate for a latest frame captured by the camera device with respect to the updated model.”).
88 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi and Sideris) to incorporate the teachings of Sucar to provide a clearer method to look at each type of iteration related to the camera pose(s) to update the global representation, utilizing the camera poses and the determination of the iterations provided by the teachings of Rochette in view of Taguchi and Sideris. Doing so would improve the camera pose estimate and the model of the environment as stated by Sucar ([0045] recited).
89 Regarding claim 23, Rochette in view of Taguchi and Sideris teaches the method of claim 21, wherein progressively building the global primitive-based representation of the video includes at each iteration (see claims 1 and 21 rejections above): but does not explicitly teach densifying a current global primitive-based representation of the video.
90 Sucar teaches densifying a current global primitive-based representation of the video ([0009] reciting “In this way, a portion of the image data can be selectively obtained for optimising the model, for example rather than using all the image data obtained.”; [0045] reciting “For example, the model may be a neural network configured to map a spatial coordinate corresponding to a location in the environment to a photometric value and a volume density value associated with the location, the volume density value being used to derive a depth value at the location. The rendered image data represents a rendered image portion corresponding to a portion of the environment observed.”; [0049] reciting “In such cases, the image data may be considered to be video data and the camera device may be considered to be a video camera.”; [0054] reciting “In examples, the model 104 can map a given spatial coordinate within the environment to a photometric value and a volume density value. This therefore allows 3D representations of various resolutions to be obtained using the model 104, contrary to voxel and point-cloud representations of an environment, which have a fixed resolution.”; [0073] reciting “For example, the set of volume density values may be transformed into a set of occupancy probabilities, o.sub.i, representing a probability that an object is occupying each of the set of spatial coordinates 126a-c.”).
91 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi and Sideris) to incorporate the teachings of Sucar to provide a method that determines and utilizes the density to be used for the specific global representations of the video provided by Rochette in view of Taguchi and Sideris. Doing so would improve the camera pose estimate and the model of the environment as stated by Sucar ([0045] recited).
92 Claim 40 has similar limitations as of claim 22, therefore it is rejected under the same rationale as claim 22.
93 Claim(s) 25-27 and 42 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claims 1 and 29, further in view of Sajjadi et al. (US 20240169662 A1).
94 Regarding claim 25, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), and although Rochette could teach wherein the view synthesis is performed for a virtual reality application ([Abstract] reciting “We show how our approach can be used for motion transfer between individuals; novel view synthesis of individuals captured from just a single camera; to synthesize individuals from any virtual viewpoint;”), prior art from Sajjadi can teach the limitation further.
95 Sajjadi teaches wherein the view synthesis is performed for a virtual reality application ([0040] reciting “In an example, image view synthesis framework 100 can be employed in the creation of virtual reality (VR) and augmented reality (AR) experiences. Generating realistic views of a scene from various angles can help provide a realistic and immersive experience for users. Framework 100 can be used to generate these views from an inexpensive set of source images, potentially reducing the amount of data or other resources used to create a realistic VR or AR environment, thereby improving the efficiency of such systems.”; [0067] reciting “For example, image view synthesis model 106 can learn an implicit pose space of camera poses using a pose estimator model.”).
96 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sajjadi to provide a clearer to method to utilize the view synthesis methods of the camera poses from Rochette in view of Taguchi in a virtual reality setting. Doing so would potentially reduce the amount of data or other resources used to create a realistic VR or AR environment as stated by Sajjadi ([0040] recited).
97 Regarding claim 26, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), but does not explicitly teach wherein the view synthesis is performed for an augmented reality application.
98 Sajjadi teaches wherein the view synthesis is performed for an augmented reality application ([0040] reciting “In an example, image view synthesis framework 100 can be employed in the creation of virtual reality (VR) and augmented reality (AR) experiences. Generating realistic views of a scene from various angles can help provide a realistic and immersive experience for users. Framework 100 can be used to generate these views from an inexpensive set of source images, potentially reducing the amount of data or other resources used to create a realistic VR or AR environment, thereby improving the efficiency of such systems.”; [0067] reciting “For example, image view synthesis model 106 can learn an implicit pose space of camera poses using a pose estimator model.”).
99 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sajjadi to provide a method to utilize the view synthesis methods of the camera poses from Rochette in view of Taguchi in an augmented reality setting. Doing so would potentially reduce the amount of data or other resources used to create a realistic VR or AR environment as stated by Sajjadi ([0040] recited).
100 Regarding claim 27, Rochette in view of Taguchi teaches the method of claim 1 (see claim 1 rejection above), but does not explicitly teach wherein the view synthesis is performed for a robotics application.
101 Sajjadi teaches wherein the view synthesis is performed for a robotics application ([0074] reciting “Image view synthesis framework 100 can pass output image 112 to other downstream systems. Image view synthesis framework 100 can pass output image 112 to a robotics perception system. Image view synthesis framework 100 can pass output image 112 to an augmented reality or virtual reality system.”).
102 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sajjadi to provide a method to utilize the view synthesis methods of the camera poses from Rochette in view of Taguchi in a robotics application setting. Doing so would potentially reduce the amount of data or other resources used to create something like a realistic VR or AR environment as stated by Sajjadi ([0040] recited).
103 Regarding claim 42, Rochette in view of Taguchi teaches the system of claim 29 (see claim 29 rejection above), and although Rochette could teach wherein the view synthesis is performed for one of: a virtual reality application, an augmented reality application, a robotics application, or a 3D content creation application ([Abstract] reciting “We show how our approach can be used for motion transfer between individuals; novel view synthesis of individuals captured from just a single camera; to synthesize individuals from any virtual viewpoint;”; [Page 1; Section I] reciting “Disentangling shape and appearance of an image is fundamental for general 3D understanding and lies at the heart of our idea. If we focus on synthesizing novel views of humans, the localization of human joints in the three-dimensional space can be seen as a first step towards human body shape estimation.”), prior art from Sajjadi can teach the limitation further.
104 Sajjadi teaches wherein the view synthesis is performed for one of: a virtual reality application, an augmented reality application ([0040] reciting “In an example, image view synthesis framework 100 can be employed in the creation of virtual reality (VR) and augmented reality (AR) experiences. Generating realistic views of a scene from various angles can help provide a realistic and immersive experience for users. Framework 100 can be used to generate these views from an inexpensive set of source images, potentially reducing the amount of data or other resources used to create a realistic VR or AR environment, thereby improving the efficiency of such systems.”; [0067] reciting “For example, image view synthesis model 106 can learn an implicit pose space of camera poses using a pose estimator model.”), a robotics application ([0074] reciting “Image view synthesis framework 100 can pass output image 112 to other downstream systems. Image view synthesis framework 100 can pass output image 112 to a robotics perception system. Image view synthesis framework 100 can pass output image 112 to an augmented reality or virtual reality system.”), or a 3D content creation application.
105 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sajjadi to provide a method to utilize the view synthesis methods of the camera poses from Rochette in view of Taguchi in other settings like virtual and/or augmented reality as well as a robotics application setting, and can be other than a 3d application taught by Rochette in view of Taguchi. Doing so would potentially reduce the amount of data or other resources used to create something like a realistic VR or AR environment as stated by Sajjadi ([0040] recited).
106 Claim(s) 47 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rochette, G., Russell, C., & Bowden, R. (2021, December). Human pose manipulation and novel view synthesis using differentiable rendering. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE. (hereinafter Rochette) in view of Taguchi et al. (US 20140003705 A1) as of claim 43, further in view of Sucar et al. (US 20240005597 A1).
107 Regarding claim 47, Rochette in view of Taguchi teaches the non-transitory computer-readable media of claim 43 (see claim 43 rejection above), but does not explicitly teach wherein at each iteration the relative camera pose is learned for the corresponding one of the plurality of pairs of sequential frames and the relative camera pose is used with the corresponding one of the plurality of pairs of sequential frames to update the global primitive-based representation of the video.
108 Sucar teaches wherein at each iteration the relative camera pose is learned for the corresponding one of the plurality of pairs of sequential frames and the relative camera pose is used with the corresponding one of the plurality of pairs of sequential frames to update the global primitive-based representation of the video ([0014] reciting “Selecting the set of pixels based on the loss probability distribution for example allows pixels to be sampled based on how useful they are likely to be in updating the model (e.g. how likely they are to correspond to parts of the environment with a large amount of detail and/or that are insufficiently represented by the model).”; [0045] reciting “Based on the loss, at least the camera pose estimate and the model are jointly optimised to generate an update to the camera pose estimate and an update to the model. This approach for example allows for a learning of the environment so as to iteratively improve the camera pose estimate and the model of the environment. Optimising the model in this manner for example improves the accuracy of the 3D representation of the environment generated using the model. This joint optimisation may be applied in a SLAM system in which, in parallel to the joint optimisation, a tracking system continuously optimises a camera pose estimate for a latest frame captured by the camera device with respect to the updated model.”).
109 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Rochette in view of Taguchi) to incorporate the teachings of Sucar to provide a method to look at each type of iterations related to the camera pose(s) to update the global representation, utilizing the camera poses and global representations provided by the teachings of Rochette in view of Taguchi. Doing so would improve the camera pose estimate and the model of the environment as stated by Sucar ([0045] recited).
Conclusion
110 Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY TRAN LE whose telephone number is (571)272-5680. The examiner can normally be reached Mon-Thu: 7:30am-5pm; First Fridays Off; Second Fridays: 7:30am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHNNY T LE/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights