Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
35 USC § 101
Claims 1 – 13 are considered to be patent eligible under 35 USC § 101.
Allowable Subject Matter
Claims 2-4 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-8 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As per claim 5:
As per the limitation, “wherein in a case where, among a plurality of regions in which corresponding ones of the plurality of first 3D models corresponding to each of the plurality of subjects are present in a virtual space, a first region is present that overlaps by more than a predetermined level with a second region in which the second 3D model is present in the virtual space, the virtual viewpoint image is generated by arranging the second 3D model corresponding to the specific subject at a position of the first region instead of the first 3D model corresponding to the first region, and is output.” The claim limitation is attempting to provide a condition, but it is not clear what happens when the condition fails. That is what happens when all of the limitation fails? What condition occurs when all the limitation fails?
As per claim 6:
As per the limitation, “wherein in a case where the second 3D model corresponding to the specific subject is not generated, the virtual viewpoint image is generated using the second 3D model corresponding to the specific subject generated in the past, and is output.” The claim limitation is attempting to provide a condition, but it is not clear what happens when the condition fails. That is what happens when all of the limitation fails? What condition occurs when all the limitation fails?
The examiner notes the dependent claim 7 is indefinite due to depending from indefinite claim 6.
As per claim 8:
As per the limitation, “wherein in a case where the second 3D model corresponding to the specific subject is not generated and the first 3D models of the specific subject are generated, information indicating an anomaly is output.” The claim limitation is attempting to provide a condition, but it is not clear what happens when the condition fails. That is what happens when all of the limitation fails? What condition occurs when all the limitation fails?
Claims 5-8 are considered to be so indefinite that prior art can’t be applied.
Claim Rejections - 35 USC § 102
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 9-12 are rejected under 35 U.S.C. 102(a)(1) or 102(a)(2) as being anticipated by Su et al. (RobustFusion: Human Volumetric Capture with Data-Driven Visual Cues Using a RGBD Camera) as cited in an 892.
Regarding claim 1, Su teaches an image generation device comprising (See Page 255 – 256, 6 Experiment “We run our experiments on a PC with a NVIDIA GeForce GTX TITAN X GPU and an Intel Core i7-7700K CPU.”):
one or more processors (See Page 255 – 256, 6 Experiment “We run our experiments on a PC with a NVIDIA GeForce GTX TITAN X GPU and an Intel Core i7-7700K CPU.”); and
one or more memories that store a computer-readable instruction for causing, when executed by the one or more processors, the one or more processors to (See Page 255 – 256, 6 Experiment “We run our experiments on a PC with a NVIDIA GeForce GTX TITAN X GPU and an Intel Core i7-7700K CPU.” NVIDIA GeForce GTX TITAN X GPU has RAM that has computer-readable instructions in the RAM.):
obtain (i) a plurality of first three-dimensional (3D) models, of corresponding ones of a plurality of subjects, generated through a first method on the basis of shooting a predetermined region in which the plurality of subjects are present (See page 248, 1 Introduction, “We propose a robust human volumetric capture method, which is the first to
embrace various data-driven visual cues under the monocular setting without
pre-scanned template, achieving significant superiority to state-of-the-arts.
– To eliminate the tedious self-scanning constraint, we propose a novel optimization
pipeline to combine data-driven occupancy representation with volumetric
fusion only using the front-view input.”), and (ii) a second 3D model that is based on posture information of a specific subject, among the plurality of subjects, present in the predetermined region during the shooting, the second 3D model corresponding to the specific subject and being generated through a second method different from the first method (See page 248, 1 Introduction, “We propose an effective robust performance capture scheme with human pose, shape and parsing priors, which can handle challenging human motions with reinitialization ability.” See Fig 4. Also see, page 252-253, 5 Robust Performance Capture “As illustrated in Fig. 4, a novel performance capture scheme is proposed to track
challenging human motions robustly with re-initialization ability with the aid of
reliable data-driven visual cues.
Initialization. Note that the final complete model from Sect. 4 provides a
reliable initialization for both the human motion and the utilized visual priors.
To this end, before the tracking stage, we first re-sample the sparse ED nodes
{xi} on the mesh to form a non-rigid motion field, denoted as G. Besides, we rig the mesh with the output pose parameters θ0 from its embedded SMPL
model in Sect. 4 and transfer the SMPL skinning weights to the ED nodes {xi}.
Then, for any 3D point vc in the capture volume, let ˜vc and ˆvc denote the warped
positions after the embedded deformation and skeletal motion, respectively. Note
that the skinning weights of vc for the skeletal motion are given by the weighted
average of the skinning weights of its knn-nodes. Please refer to [30,61,66] for
more detail about the motion formulation. To initialize the pose prior, we apply
OpenPose [6] on the RGBD image to obtain the 2D and lifted 3D joint positions,
denoted as P2D l and P3D l , respectively, with a detection confidence Cl. Then, we
find the closest vertex from the watertight mesh to P3D l , denoted as Jl, which is
the associated marker position for the l-th joint. To utilize the semantic visual
prior, we apply the light-weight human parsing method [70] to the triggered
RGB image to obtain a human parsing image L. Then, we project each ED node
xi into L to obtain its initial semantic label li. After the initialization, inspired
by [19,64], we propose to optimize the motion parameters G and θ in an iterative
flip-flop manner, so as to fully utilize the rich motion prior information of the
visual cues to capture challenging motions.”); and
output a virtual viewpoint image generated on the basis of (i) the first 3D model of a subject, among the plurality of subjects, that is different from the specific subject, and (ii) the second 3D model corresponding to the specific subject (See Fig. 5).
Regarding claim 9, Su teaches the image generation device according to claim 1,
wherein the first method is a volumetric capture method (See page 249-250, 3 Overview, “RobustFusion marries volumetric capture to various data-driven human visual
cues, which not only eliminates the tedious self-scanning constraint but also captures challenging human motions robustly handling the severe tracking failures
or disappear-reoccur scenarios. As illustrated in Fig. 1, our approach takes
a RGBD video from Kinect v2 as input and generates 4D meshes, achieving
considerably more robust results than previous methods. Similar to [61,66], we
utilize TSDF [8] volume and ED model [49] for representation.”), and the second method is a motion capture method (See page 6.2 Evaluation, “The result in Fig. 9 (b) is still imperfect because only the skeletal pose is optimized during completion optimization and only SMPL-based ED-graph is adopted for motion tracking.” See page 250, “The core of our pipeline is to solve the hybrid motion parameters from the canonical frame to current camera view. We propose a robust tracking scheme which utilizes the reliable visual priors to optimize both the skeletal and non-rigid motions in an iterative flip-flop manner. Our scheme can handle challenging motions with the reinitialization ability.”).
Regarding claim 10, Su teaches the image generation device according to claim 1,
wherein the first 3D models are generated by generating a three-dimensional shape of each of the plurality of subjects on the basis of shooting of the predetermined region by a plurality of cameras and then applying a texture obtained from the shooting to the three-dimensional shape (See page 257, “Fig. 9. Evaluation of model completion. (a–c) are the results without completion, with completion only using skeleton optimization and SMPL-based ED-graph, with our full model completion, respectively. Each triple includes the output mesh, corresponding ED-graph and the overlaid tracking geometry/texture results for a following frame.”.)
Regarding claim 11, Su teaches the image generation device according to claim 9,
wherein the second 3D model is generated by generating a bone model of the specific subject from posture information that is based on information obtained from a plurality of sensors, and then deforming a computer graphics (CG) model to the bone model (See page 257, “Fig. 9. Evaluation of model completion. (a–c) are the results without completion, with completion only using skeleton optimization and SMPL-based ED-graph, with our full model completion, respectively. Each triple includes the output mesh, corresponding ED-graph and the overlaid tracking geometry/texture results for a following frame.”.).
Claim 12 and 13 recite similar limitations to that of claim 1 and thus is rejected under similar rationale as detailed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Alexiadis et al., An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing, as cited in an 892, abstract, “The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors’ calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies. “
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT J CRADDOCK whose telephone number is (571)270-7502. The examiner can normally be reached Monday - Friday 10:00 AM - 6 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT J CRADDOCK/Primary Examiner, Art Unit 2618
Prosecution Insights