DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 10, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Choi (US Patent 11158104), in view of Montes et al. (US Pub. 2024/0144593), hereinafter Montes.
Regarding claim 1, Choi discloses a method of estimating a three-dimensional (3D) skull model representative of a patient's facial bone structure (Fig. 5; Column 17, lines 1-15: machine learning engine 1010 may generate a skull surface 1012, which is further output to a model editor 1015. For example, the model editor 1015 may display the constructed 3D skull surface on a user interface 1016, e.g., similar to 1005 in FIG. 5. A user, such as an animator, etc., may edit the generated skull surface by submitting an input 1018 via the user interface, e.g., by shifting, modifying, smoothing, adjusting, and/or the like certain parts of the generated skull surface 1012. For example, the user may directly interact with a graphic user interface (GUI) to modify the skull topology, such as enlarging an eye socket, lowering the curve of cheekbone, and/or the like. For another example, the user may enter specific parameters to change the size, shape, and/or position coordinates of a specific bone in the skull), the method comprising: receiving, or generating from a facial scan, a 3D skin model representative of an outer surface of the patient's head (Fig. 5; Column 11, lines 6-12: number of facial scans 501a-n may be obtained, e.g., see 108 in FIG. 1, and used for anatomical data building. Each facial scan may include an image of the skin surface of the live actor, e.g., skin surface data 504, and a set of facial muscle sensing data 503 accompanying the skin surface data 504; Column 16, line 50-Column 17, line 15: the tissue depth array 1002 may include a plurality of muscle points (fiducial markers) and the corresponding average tissue thickness underneath the fiducial markers. A plurality of bone attachment points on the skull may then be derived based on the fiducial markers on skin surface and the average tissue thickness at 1003. The plurality of bone attachment points may be used to construct, e.g., via interpolation, a 3D skull surface 1005…Facial scan 1001 that includes the skin topology and generic tissue dataset 1002 including a tissue depth array may be input to a machine learning engine 1010, which learns the underneath skull structure corresponding to the facial scan 1001. In one implementation, the generic tissue dataset 1002 may be selected based on specific race, ethnicity, age, or the body mass index of the live actor…machine learning engine 1010 may generate a skull surface 1012, which is further output to a model editor 1015. For example, the model editor 1015 may display the constructed 3D skull surface on a user interface 1016, e.g., similar to 1005 in FIG. 5. A user, such as an animator, etc., may edit the generated skull surface by submitting an input 1018 via the user interface, e.g., by shifting, modifying, smoothing, adjusting, and/or the like certain parts of the generated skull surface 1012); generating a combined mesh comprising the 3D skin model and a candidate 3D skull model (Column 10, line 64-Column 11, line 12: FIG. 5 provides an example diagram illustrating building the anatomical model of the specific live actor for the animation pipeline shown in FIG. 1, according to one embodiment described herein. The muscle parameter aggregation 505 and the muscle topology approximation 515 may be a combined module, or separate modules, which may serve similar functions as the pseudo-muscle generation 114 in FIG. 1. The skull structure generation module 550 may be similar to module 115 in FIG. 1…the skin surface data 504 together with the reference data indicating facial tissue depth from the muscle anatomy database 519 may be fed to the skull structure generation 550 to construct a skull structure 555 specific to the live actor. For example, the muscle anatomy database 519 may provide generic facial tissue depth data including an average tissue thickness of each facial muscle); generating a reprojected mesh from a trained machine learning model using the combined mesh as input (Fig. 5; Column 17, lines 1-15: machine learning engine 1010 may generate a skull surface 1012, which is further output to a model editor 1015. For example, the model editor 1015 may display the constructed 3D skull surface on a user interface 1016, e.g., similar to 1005 in FIG. 5. A user, such as an animator, etc., may edit the generated skull surface by submitting an input 1018 via the user interface, e.g., by shifting, modifying, smoothing, adjusting, and/or the like certain parts of the generated skull surface 1012. For example, the user may directly interact with a graphic user interface (GUI) to modify the skull topology, such as enlarging an eye socket, lowering the curve of cheekbone, and/or the like. For another example, the user may enter specific parameters to change the size, shape, and/or position coordinates of a specific bone in the skull).
Choi does not explicitly disclose generating the 3D skull model by removing the 3D skin model from the reprojected mesh.
However, Montes teaches 3D head model generation (Abstract), further comprising generating the 3D skull model by removing the 3D skin model from the reprojected mesh (Paragraph [0045]: skin segmentation may generate a 3D scan from a patient's functional MRI data. As previously described, skin segmentation is the process of creating an identified subset of voxels using a governing characteristic that is shared between them. When voxel intensity is used as the governing characteristic, a process referred to as threshold segmentation may group all voxels having the same identified voxel intensity into the same segment. Different body tissues and materials generally have different voxel intensity thresholds that capture them. This makes it possible to set a voxel intensity to capture bone, air, soft tissue independently. For example, as shown in FIG. 8, a single threshold may be used to extract a complete skin segmentation of the head of a patient from the patient's fMRI data. It is generally not possible to extract a complete skin segmentation using just one threshold since each tissue type (e.g., bone, air, soft tissue) has a distinct voxel intensity. When one threshold is used, the raw data is not normalized and its precision varies from one point in space to another. The data gets more noisy and less precise as you approach the soft tissue of the nose, for example. This results in a skin segmentation that creates holes that prevent the creation of realistic 3D representations). Montes teaches that this will allow for creation of more complete or realistic representation of the skull and face of a patient (Paragraph [0009]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Choi with the features of above as taught by Montes so as to allow for creation of more complete or realistic representation of the skull and face of a patient as presented by Montes.
Regarding claim 2, Choi, in view of Montes teaches the method of claim 1, Choi discloses wherein the 3D skin model is generated from the facial scan, and wherein the facial scan is performed by capturing a plurality of two-dimensional (2D) of the patient's face at different orientations with respect to the patient's face (Fig. 5; Column 11, lines 6-11 and 56-67: number of facial scans 501a-n may be obtained, e.g., see 108 in FIG. 1, and used for anatomical data building. Each facial scan may include an image of the skin surface of the live actor, e.g., skin surface data 504, and a set of facial muscle sensing data 503 accompanying the skin surface data 504… a subset of facial muscles from the muscle polygon topology 510 may be selected and pseudo-muscles are generated to approximate the subset of facial muscles in positions such that each pseudo-muscle may represent or substitute a facial muscle the topology 510. For example, based on generic human anatomical knowledge, some facial muscles may have “subunits,” e.g., a portion of a muscle may act independently from another portion of the muscle depending on how it is innervated by the nervous system. In this case, the “subunit” of the muscle can be selected. For another example, the directions of skin movements of the live actor may be observed).
Regarding claim 3, Choi, in view of Montes teaches the method of claim 1, Choi discloses wherein generating the combined mesh comprises registering the 3D skin model with the candidate 3D skull model, and wherein the registering utilizes facial landmarks to estimate position and orientation of the candidate 3D skull model with respect to the 3D skin model (Column 13, lines 37-67: the skull structure generation 550 may then use the reference facial soft tissue thickness data and the skin surface data 504 to derive a skull structure. For example, for each fiducial marker position 504 from the facial scans 510a-n, the skull structure generation 550 may map the respective fiducial marker to a soft tissue from a tissue array in the soft tissue thickness dataset, and may then compute the position of a corresponding spot on the skull where the soft tissue is attached to by offsetting the position of the fiducial marker by the respective thickness of the soft tissue. In this way, the skull structure may be reconstructed by interpolating all the computed positions of spots on the skull…the skull structure generation 550 may further assess the pseudo-muscle topology 515 from the topology approximation 515 to further incorporate personalized anatomical data into skull structure generation. For example, when mapping a respective fiducial marker in the skin surface data 504 to a soft tissue from a tissue array in the soft tissue thickness dataset, the pseudo-muscle topology 515 may be used to provide an estimate of the shape and position of the muscle underneath the position of the respective fiducial marker on the skin surface).
Regarding claim 4, Choi, in view of Montes teaches the method of claim 1, Choi discloses wherein using the combined mesh as input comprises generating an input vector comprising a latent space representation of the combined mesh, and wherein generating the reprojected mesh comprises projecting the latent space representation of the combined mesh into a data space representation (Column 7, line 63-Column 8, line 15: the generated skull structure topology from skull structure generation 115, the pseudo-muscle topology from the pseudo-muscle generation module 114, together with the data bundles representing the time-varying vectors of muscle parameters, skin representation, joint representation, and/or the like over a bundle time period, may be input to the machine learning model 118. Based on parameters in the data bundles such as parameters of the muscles, strains, joints, and/or the like, skull parameters from the skull topology, static muscle parameters from the pseudo-muscle topology, the machine learning model 118 generates a predicted skin surface representation (e.g., the visible facial expression such as “smile,” “frown,” etc.). In this way, the machine learning model 118 can learn a transformation between parameters of the muscles, strains, joints, and/or the like and the skin surface representation of actor A through a training dataset in the form of data bundles representing scan results 108 from the actor A).
Regarding claim 6, Choi, in view of Montes teaches the method of claim 1, Choi discloses further comprising receiving additional data sets for input to the machine learning model, the additional data sets comprising one or more of panoramic X-ray scan data, cephalometric X-ray scan data, magnetic resonance imaging scan data, partial cone beam computed tomography scan data, or articulation capture data (Column 6, lines 27-33: obtain specific skull surface for a specific live actor based on invasive procedures, such as magnetic resonance imaging (MRI) or X-ray scans. Such invasive methods can be both expensive and inconvenient and can hardly be implemented on a massive scale when a large number of live actors are involved in film production).
Regarding claim 10, Choi, in view of Montes teaches a system for estimating a three-dimensional (3D) skull model representative of a patient's facial bone structure, the system comprising: a memory (Choi: Column 22, lines 58-67: computer system 1300 also includes a main memory 1306, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1302 for storing information and instructions to be executed by the processor 1304. The main memory 1306 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 1304); and a processing device operatively coupled to the memory (Choi: Column 22, lines 58-67: computer system 1300 also includes a main memory 1306, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1302 for storing information and instructions to be executed by the processor 1304. The main memory 1306 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 1304), wherein the processing device is configured to perform the method of claim 1 (See claim 1).
Regarding claim 11, Choi, in view of Montes teaches a non-transitory machine-readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to perform the method of claim 1 (See claim 1; Choi: Column 22, lines 30-40: code may also be provided and/or carried by a transitory computer readable medium, e.g., a transmission medium such as in the form of a signal transmitted over a network).
Allowable Subject Matter
Claims 5 and 7-9 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim 5 would be allowable, in independent form, over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a method comprising receiving aligned intraoral scan data representative of the patient's upper and lower dental arches, wherein generating the reprojected mesh based at least in part on the aligned intraoral scan data; and non-rigidly deforming the 3D skull model to conform its shape and alignment with the upper and lower dental arches of the aligned intraoral scan data, as presented in the environment of the remaining limitations of claim 5. It is noted that Choi, in view of Montes teaches the method of claim 1. However, Choi, in view of Montes fails to disclose or suggest receiving aligned intraoral scan data representative of the patient's upper and lower dental arches, wherein generating the reprojected mesh based at least in part on the aligned intraoral scan data; and non-rigidly deforming the 3D skull model to conform its shape and alignment with the upper and lower dental arches of the aligned intraoral scan data.
Claim 7 would be allowable, in independent form, over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a method comprising integrating the 3D skull model with one or more data sets or processes for use in visualization of a dental treatment plan for the patient, as presented in the environment of the remaining limitations of claim 7. It is noted that Choi, in view of Montes teaches the method of claim 1. However, Choi, in view of Montes fails to disclose or suggest integrating the 3D skull model with one or more data sets or processes for use in visualization of a dental treatment plan for the patient.
Claim 8 would be allowable, in independent form, over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a method comprising wherein the volumetric mesh can be used with physically-based simulation techniques to simulate deformations and predict changes to soft tissue of the virtual patient's face responsive to a dental treatment plan, as presented in the environment of the remaining limitations of claim 8. It is noted that Choi, in view of Montes teaches the method of claim 1, further comprising: generating a volumetric mesh representative of soft tissue of a virtual patient's head based on the 3D skin model and the 3D skull model. However, Choi, in view of Montes fails to disclose or suggest wherein the volumetric mesh can be used with physically-based simulation techniques to simulate deformations and predict changes to soft tissue of the virtual patient's face responsive to a dental treatment plan.
Claim 9 depends from claim 8 and would therefore be allowable for the same reasons as above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Singh et al. (US Pub. 2024/0169635) teaches latent space facial modelling.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights