DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
2. Acknowledgement is made of amendment filed on July 22, 2026, in which claims 2, 5, 7, 9, 12, 15, 18 and 19 are amended, and claims 2-21 are still pending.
Response to Arguments
3. Applicant's arguments, filed on July 22, 2026, with respect to Claims 2-21 have been fully considered and they are not persuasive.
4. With regards to arguments for independent claims 2, 12 and 19, applicants argue that Sutton et al. (US 2019/0213779 A1) and Stefan et al. (US 7,783,099 B1) fail to disclose process a combined input, comprising a first input of the 3D model of the current condition of the body part of the person and a second input of patient specific parameters corresponding to a proposed medical procedure to be performed on the body part of the person, using a trained machine learning model to generate a 3D model of a predicted post treatment condition of the body part of the person. However, the examiner respectfully disagrees that Stutton and Stefan fail to disclose the arguments regarding claims 2, 12 and 19, since in Sutton et al. (US 2019/0213779 A1) teaches (“a cropping program is utilized to reshape the DICOM dataset, the new cropped dataset is converted to an alternate signal intensity format through a bias field correction algorithm, and the new alternate signal intensity format dataset (that is, the bias field corrected dataset) is then compared to a previously prepared expert manually trained ground truth dataset where myocardial signal intensity elements (that fall within “ground truth” parameters) are selected in the bias field corrected dataset. Selected voxels (in the bias field corrected dataset) are then assigned to coordinate system in 3D model format and output as both volumetric dataset and 3D file format. Still referring to FIG. 6, certain specific details of a process for creating a trained machine dataset according to this embodiment are as follows: Block 601—DICOM dataset: voxel based data with each voxel representing a signal intensity value at a fixed coordinate position within the dataset.” [0059-0060]) Stutton teaches the trained machine dataset in 3D model. Also, in Stefan et al. (US 7,783,099 B1) teaches (“the preoperative candidate may review the forecasted 3-D images, either alone or together with a surgeon, and select the best overall aesthetic forecasted 3-D image that meets the desires of the patient, and the surgical guidelines, given the preoperative candidate's overall body condition. … once a forecasted 3-D image is selected, the selected forecasted 3-D image is converted from a 3-D image to a format that a scanner can read and measure. In various exemplary embodiments, this is accomplished via a data converter. The data converter takes the forecasted 3-D image, which is in 3-D format, such as, for example a VRML format, and converts the forecasted 3-D image into a format that the scanner can interpret.” col. 5, lines 49-62 “the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 have each been converted from the scan image format to a 3-D image format.” col. 7, lines 11-14) Stefan teaches converts the forecasted 3-D image into a format that the scanner can interpret, and the postoperative breast augmentation patient images converted from the scan image format to a 3-D image format. The arguments also moot in view of the new grounds of rejections regarding claims 2, 12 and 19, since in Spilker et al. (US 2017/0372096 A1) teaches (“creating a three-dimensional model representing at least a portion of anatomy based on patient-specific data regarding a geometry of a patient's anatomy; determining a geometry associated with at least one of a plurality of treatment options for treating at least a portion of the patient's anatomy;” claim 29 “determining an original patient geometry based on the three-dimensional model; and determining a proposed treated geometry associated with the at least one of the plurality of treatment options, wherein the proposed treated geometry is based, at least in part, on the geometry associated with the at least one of the plurality of treatment options, wherein the modification of the three-dimensional model is based on the geometry of the patient's anatomy, the proposed treated geometry, or a combination thereof.” claim 31) Spilker teaches the modification of the three-dimensional model is based on the geometry of the patient's anatomy, the proposed treated geometry, or a combination thereof. While Spilker teaches process a combined input, comprising a first input of the 3D model of the current condition of the body part of the person and a second input of patient specific parameters corresponding to a proposed medical procedure to be performed on the body part of the person, Stutton teaches using a trained machine learning model, and Stefan teaches generate a 3D model of a predicted post treatment condition of the body part of the person. Therefore, the combination of Spilker, Stutton and Stefan teaches the arguments of the limitations for claims 2, 12 and 19 as they are recited.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
8. Claim(s) 2, 3, 6, 8, 11-13, 16, 17, 19 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sutton et al. (US 2019/0213779 A1) in view of Spilker et al. (US 2017/0372096 A1) and Stefan et al. (US 7,783,099 B1).
9. With reference to claim 2, Sutton teaches A computer system comprising: a memory; and a processor operatively coupled to the memory (“The computer system 1100 may include a processor 1102 (e.g., a central processing unit (CPU), a graphics processing unit (GPU, or both), a main memory 1104 and a static memory 1106, which communicate with each other via a bus 1108.” [0099]) Sutton also teaches the processor is to: obtain a two-dimensional (2D) image depicting a current condition of a body part of a person; (“step 702 comprises receiving, by a processing system including a processor, an input three-dimensional dataset comprising a first plurality of two-dimensional images of all or a portion of a subject.” [0062] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084]) Sutton further teaches generate a three-dimensional (3D) model of the current condition of the body part of the person from the 2D image; (“step 902 comprises obtaining a trained three-dimensional dataset comprising a first plurality of two-dimensional images covering a first volume of a training subject, wherein the trained three-dimensional dataset is based upon one or more first magnetic resonance imaging scans of the training subject. Next, step 904 comprises obtaining an input three-dimensional dataset comprising a second plurality of two-dimensional images covering a second volume of a patient, wherein the input three-dimensional dataset is based upon one or more second magnetic resonance imaging scans of the patient. Next, step 906 comprises cropping the input three-dimensional dataset to generate a cropped three-dimensional dataset comprising a third plurality of two-dimensional images covering a third volume of the patient, wherein the third volume is smaller than the second volume. … step 910 comprises generating a segmented three-dimensional dataset comprising a fifth plurality of two-dimensional images, … As described herein are mechanisms for providing 3D segmented organ models from raw MRI data (e.g., for use as a pre-surgical tool).” [0064-0065] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084]) Sutton teaches using a trained machine learning model (“a cropping program is utilized to reshape the DICOM dataset, the new cropped dataset is converted to an alternate signal intensity format through a bias field correction algorithm, and the new alternate signal intensity format dataset (that is, the bias field corrected dataset) is then compared to a previously prepared expert manually trained ground truth dataset where myocardial signal intensity elements (that fall within “ground truth” parameters) are selected in the bias field corrected dataset. Selected voxels (in the bias field corrected dataset) are then assigned to coordinate system in 3D model format and output as both volumetric dataset and 3D file format. Still referring to FIG. 6, certain specific details of a process for creating a trained machine dataset according to this embodiment are as follows: Block 601—DICOM dataset: voxel based data with each voxel representing a signal intensity value at a fixed coordinate position within the dataset.” [0059-0060]) Stutton teaches the trained machine dataset in 3D model.
PNG
media_image1.png
733
556
media_image1.png
Greyscale
Sutton does not explicitly teach process a combined input, comprising a first input of the 3D model of the current condition of the body part of the person and a second input of patient specific parameters corresponding to a proposed medical procedure to be performed on the body part of the person, using a trained machine learning model to generate a 3D model of a predicted post treatment condition of the body part of the person; and output a view of the 3D model of the predicted post treatment condition of the body part of the person. This is what Spilker teaches. Spilker teaches process a combined input, comprising a first input of the 3D model of the current condition of the body part of the person and a second input of patient specific parameters corresponding to a proposed medical procedure to be performed on the body part of the person, (“creating a three-dimensional model representing at least a portion of anatomy based on patient-specific data regarding a geometry of a patient's anatomy; determining a geometry associated with at least one of a plurality of treatment options for treating at least a portion of the patient's anatomy;” claim 29 “determining an original patient geometry based on the three-dimensional model; and determining a proposed treated geometry associated with the at least one of the plurality of treatment options, wherein the proposed treated geometry is based, at least in part, on the geometry associated with the at least one of the plurality of treatment options, wherein the modification of the three-dimensional model is based on the geometry of the patient's anatomy, the proposed treated geometry, or a combination thereof.” claim 31) Spilker teaches the modification of the three-dimensional model is based on the geometry of the patient's anatomy, the proposed treated geometry, or a combination thereof. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Spilker into Sutton, in order to provide a patient specific 3D representation of the expected treatment outcome.
The combination of Sutton and Spilker does not explicitly teach generate a 3D model of a predicted post treatment condition of the body part of the person; and output a view of the 3D model of the predicted post treatment condition of the body part of the person. These are what Stefan teaches. Stefan teaches generate a 3D model of a predicted post treatment condition of the body part of the person; (“the preoperative candidate may review the forecasted 3-D images, either alone or together with a surgeon, and select the best overall aesthetic forecasted 3-D image that meets the desires of the patient, and the surgical guidelines, given the preoperative candidate's overall body condition. … once a forecasted 3-D image is selected, the selected forecasted 3-D image is converted from a 3-D image to a format that a scanner can read and measure. In various exemplary embodiments, this is accomplished via a data converter. The data converter takes the forecasted 3-D image, which is in 3-D format, such as, for example a VRML format, and converts the forecasted 3-D image into a format that the scanner can interpret.” col. 5, lines 49-62 “the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 have each been converted from the scan image format to a 3-D image format.” col. 7, lines 11-14) Stefan teaches converts the forecasted 3-D image into a format that the scanner can interpret, and the postoperative breast augmentation patient images converted from the scan image format to a 3-D image format. Stefan also teaches output a view of the 3D model of the predicted post treatment condition of the body part of the person. (“the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 are profiled and displayed. It should be appreciated that, as shown in FIG. 4, the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 have each been converted from the scan image format to a 3-D image format.” col. 7, lines 8-14) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Stefan into the combination of Sutton and Spilker, in order to provide the patients and potential patients with accurate forecasting and modeling of expected results and outcomes.
10. With reference to claim 3, Sutton teaches the 2D image comprises a medical image of the person. (“step 702 comprises receiving, by a processing system including a processor, an input three-dimensional dataset comprising a first plurality of two-dimensional images of all or a portion of a subject.” [0062] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084])
11. With reference to claim 6, Sutton teaches the 3D model of the current condition of the body part is generated from the 2D image using differentiable rendering. (“step 902 comprises obtaining a trained three-dimensional dataset comprising a first plurality of two-dimensional images covering a first volume of a training subject, wherein the trained three-dimensional dataset is based upon one or more first magnetic resonance imaging scans of the training subject. Next, step 904 comprises obtaining an input three-dimensional dataset comprising a second plurality of two-dimensional images covering a second volume of a patient, wherein the input three-dimensional dataset is based upon one or more second magnetic resonance imaging scans of the patient. Next, step 906 comprises cropping the input three-dimensional dataset to generate a cropped three-dimensional dataset comprising a third plurality of two-dimensional images covering a third volume of the patient, wherein the third volume is smaller than the second volume. … step 910 comprises generating a segmented three-dimensional dataset comprising a fifth plurality of two-dimensional images, wherein the segmented three-dimensional dataset further comprises one or more markings indicating an anatomical structure within the third volume, and wherein the segmented three-dimensional dataset is generated via a convolutional neural network based upon the corrected three-dimensional dataset and based upon the trained three-dimensional dataset. … As described herein are mechanisms for providing 3D segmented organ models from raw MRI data (e.g., for use as a pre-surgical tool).” [0064-0065] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084])
12. With reference to claim 8, Sutton teaches the 3D model of the current condition of the body part of the person is generated from one or more of a morphable mesh, a parametric mesh model, or a parametric model. (“a cropping program is utilized to reshape the DICOM dataset, the new cropped dataset is converted to an alternate signal intensity format through a bias field correction algorithm, and the new alternate signal intensity format dataset (that is, the bias field corrected dataset) is then compared to a previously prepared expert manually trained ground truth dataset where myocardial signal intensity elements (that fall within “ground truth” parameters) are selected in the bias field corrected dataset. Selected voxels (in the bias field corrected dataset) are then assigned to coordinate system in 3D model format and output as both volumetric dataset and 3D file format.” [0059] “As described herein are mechanisms for providing a fully automated software pipeline to take raw MRI data and convert into a patient-specific 3D model (e.g., to be used for planning surgery). In one specific example, myocardium segmentation can be fully automated.” [0066] “As described herein are mechanisms that can be used in the context of surgical planning for cardiac procedures (often the heart is the most difficult organ to model because of movement).” [0070] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084])
13. With reference to claim 11, Sutton teaches the computer system comprises a mobile device of the person. (“this depicts an illustrative diagrammatic representation of a machine in the form of a computer system 1100 within which a set of instructions, when executed, may cause the machine to perform any one or more of the methods discussed herein. … The machine may comprise a server computer, a client user computer, a personal computer (PC), a tablet PC, a smart phone, a laptop computer, a desktop computer, a control system, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. It will be understood that a communication device of the subject disclosure includes broadly any electronic device that provides voice, video or data communication. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. The computer system 1100 may include a processor 1102 (e.g., a central processing unit (CPU), a graphics processing unit (GPU, or both), a main memory 1104 and a static memory 1106, which communicate with each other via a bus 1108.” [0097-0099])
14. Claim 12 is similar in scope to claim 2, and thus is rejected under similar rationale. Sutton additionally teaches A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations (“While the tangible computer-readable storage medium 1122 is shown in an example embodiment to be a single medium, the term “tangible computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “tangible computer-readable storage medium” shall also be taken to include any non-transitory medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methods of the subject disclosure.” [0103] “A machine-readable storage medium comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations,” claim 19)
15. Claim 13 is similar in scope to claim 3, and thus is rejected under similar rationale.
16. Claim 16 is similar in scope to claim 6, and thus is rejected under similar rationale.
17. Claim 17 is similar in scope to claim 8, and thus is rejected under similar rationale.
18. Claim 19 is similar in scope to claim 2, and thus is rejected under similar rationale.
19. Claim 21 is similar in scope to claim 8, and thus is rejected under similar rationale.
20. Claim(s) 4, 14 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sutton et al. (US 2019/0213779 A1), Spilker et al. (US 2017/0372096 A1) and Stefan et al. (US 7,783,099 B1), as applied to claims 2, 12 and 19 above, and further in view of Winner et al. (US 2017/0119471 A1).
21. With reference to claim 4, the combination of Sutton, Spilker and Stefan does not explicitly teach the body part comprises a body part on a face of the person. This is what Winner teaches (“The present invention uses augmented reality (AR) to generate a virtual image of breasts that is overlaid on a tracking marker 160, such that a patient can view herself with the virtual breasts 202 on her person 200 (see FIGS. 7A-7C).” [0018]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Winner into the combination of Sutton, Spilker and Stefan, in order to view an extremely accurate preview of their expected post-operation appearance that resembles looking at themselves in a full length mirror.
22. Claim 14 is similar in scope to claim 4, and thus is rejected under similar rationale.
23. With reference to claim 20, Sutton teaches the 2D image comprises a medical image of the person. (“step 702 comprises receiving, by a processing system including a processor, an input three-dimensional dataset comprising a first plurality of two-dimensional images of all or a portion of a subject.” [0062] “(1) Registration of current heart image to the example subject in an applicable database using ‘imregister’ in MATLAB. (2) Use that transformation to take the mask that was defined on the example subject back into the space of the current heart (the heart can then be cropped).” [0084])
The combination of Sutton, Spilker and Stefan does not explicitly teach the body part comprises a body part on a face of the person. This is what Winner teaches (“The present invention uses augmented reality (AR) to generate a virtual image of breasts that is overlaid on a tracking marker 160, such that a patient can view herself with the virtual breasts 202 on her person 200 (see FIGS. 7A-7C).” [0018]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Winner into the combination of Sutton, Spilker and Stefan, in order to view an extremely accurate preview of their expected post-operation appearance that resembles looking at themselves in a full length mirror.
24. Claim(s) 5, 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sutton et al. (US 2019/0213779 A1), Spilker et al. (US 2017/0372096 A1) and Stefan et al. (US 7,783,099 B1), as applied to claims 2 and 12 above, and further in view of Giraldez et al. (US 2021/0228276 A1).
25. With reference to claim 5, Sutton teaches the trained machine learning model comprises a convolutional neural network (CNN). (“FIG. 2 depicts a network architecture of a deep-CNN (convolutional neural network) for myocardial labeling according to an embodiment.” [0009] “an expert manually segments greyscale images for training a machine learning algorithm and patient-specific greyscale images are segmented by the trained machine learning algorithm.” [0093])
The combination of Sutton, Spilker and Stefan does not explicitly teach the trained machine learning model is trained using a plurality of 3D models of prior patients generated before respective surgical procedures and a plurality of 3D models of the prior patients generated after the respective surgical procedures. This is what Giraldez teaches (“training a machine learning system on a training dataset comprising a plurality of 3D models of at least parts of a plurality of patients' bodies at different times following a procedure and using the machine learning system to simulate the effects of the procedure on the patient's body.” claim 4 “following completion of the procedure, creating at least one 3D model of at least a part of the patient's body that has been affected by the procedure at at least one different time following the procedure and adding the at least one 3D model to the training dataset of the machine learning system.” claim 5 “training a machine learning system on training data comprising the effects of the procedure on a plurality of different patients' bodies, as performed by a plurality of different physicians, to generate a plurality of predictive models;” claim 33) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Giraldez into the combination of Sutton, Spilker and Stefan, in order to enable the model to predict a post treatment 3D model for a new patient.
26. With reference to claim 7, the combination of Sutton and Spilker does not explicitly teach an augmented reality apparatus, wherein the 2D image is one of a stream of 2D images, and wherein the view of the 3D model of the predicted post treatment condition of the body part is displayed in the augmented reality apparatus, the view being overlaid on the stream of 2D images without requiring placement of a physical tracking marker on the body part of the person. This is what Stefan teaches. Stefan teaches an augmented reality apparatus, wherein the view of the 3D model of the predicted post treatment condition of the body part is displayed in the augmented reality apparatus. (“Virtual surgery systems, methods, and apparatuses that provide for the prediction, evaluation, and validation of various cosmetic and reconstructive surgical procedures. The virtual surgery systems, methods, and apparatuses utilize a scanner, measurement software, queries to a postoperative patient scan catalog, conversion of a preoperative image to a Virtual Reality Modeling Language (3-D) image, grafting and scaling of chosen postoperative surgery candidates, conversion from the converted 3-D image to a scanner readable format, and measuring the forecasted 3-D image.” Abstract, lines 1-10 “the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 are profiled and displayed. It should be appreciated that, as shown in FIG. 4, the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 have each been converted from the scan image format to a 3-D image format.” col. 7, lines 8-14) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Stefan into Sutton, in order to provide the patients and potential patients with accurate forecasting and modeling of expected results and outcomes.
The combination of Sutton, Spilker and Stefan does not explicitly teach the 2D image is one of a stream of 2D images, and the view being overlaid on the stream of 2D images without requiring placement of a physical tracking marker on the body part of the person. This is what Giraldez teaches (“The simulation rendering libraries may further include simulation streaming libraries for streaming simulations complied from one or more post-operative stage patient models provided by the 2D/3D modelling logic 216 and/or complications models and simulations provided by the complication simulation logic 218. In one example, the streaming libraries provide for simulation streaming over a content streaming network configured for variable or adaptive bitrate streaming. One or more of the 2D/3D modelling logic 216, the rendering libraries 217, or the complication simulation logic 218 may also include matching logic for matching the orientation of the 3D model in an AR environment with the orientation of a patient body part in real time. The matching logic includes one or more libraries for tracking movement of a patient body part in live streaming video and automatically adjusting the 3D model object depicted in the AR environment to dynamically fit the patient's body part. Alternatively, the matching logic may match the orientation of a 2D post-operative stage patient model or a complication model to a 2D picture of digital image or a patient's body. In this example, the imaging engine 215 overlays the virtual model over the 2D photograph or digital image to augment the appearance of the photographed body part with a virtual representation of the desired procedure impacts, complication, or recovery effects.” [0078]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Giraldez into the combination of Sutton, Spilker and Stefan, in order to enable the model to predict a post treatment 3D model for a new patient.
27. Claim 15 is similar in scope to claim 5, and thus is rejected under similar rationale.
28. Claim(s) 9, 10 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sutton et al. (US 2019/0213779 A1), Spilker et al. (US 2017/0372096 A1) and Stefan et al. (US 7,783,099 B1), as applied to claims 2 and 12 above, and further in view of Sachdeva et al. (US 2007/0207437 A1).
29. With reference to claim 9, Sutton teaches the 2D image is received from a mobile device of the person. (“step 702 comprises receiving, by a processing system including a processor, an input three-dimensional dataset comprising a first plurality of two-dimensional images of all or a portion of a subject.” [0062] “this depicts an illustrative diagrammatic representation of a machine in the form of a computer system 1100 within which a set of instructions, when executed, may cause the machine to perform any one or more of the methods discussed herein. … The machine may comprise a server computer, a client user computer, a personal computer (PC), a tablet PC, a smart phone, a laptop computer, a desktop computer, a control system, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. It will be understood that a communication device of the subject disclosure includes broadly any electronic device that provides voice, video or data communication. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. The computer system 1100 may include a processor 1102 (e.g., a central processing unit (CPU), a graphics processing unit (GPU, or both), a main memory 1104 and a static memory 1106, which communicate with each other via a bus 1108.” [0097-0099])
The combination of Sutton, Spilker and Stefan does not explicitly teach a color 2D image, and wherein the 3D model of the predicted post treatment condition is a textured 3D model comprising color information. This is what Sachdeva teaches (“the system employs software simulation of changes in shape or position of craniofacial structures (e.g., teeth or jaw) on the visual appearance, e.g., smile, of the patient. Accordingly, at least one of the data sets will include normally include data regarding the surface configuration of the face and head. A commercially available digital CCD camera 28 (FIG. 1), e.g., a color or black and white digital camera available from Sony or Canon, can be used to obtain this information. Preferably, the image data is color image data.” [0066] “FIGS. 4A-4E show several screen displays from a user interface of the unified workstation that illustrate the process of texture mapping a 3D object (here, teeth) by projection of color data from a 2D photograph. After a patient's dentition is scanned, the virtual teeth and gingiva for both upper and lower arches are represented as a single surface, in the present example a triangle mesh surface. FIG. 4A shows a 2D digital photograph of teeth/gingivae 71 displayed in a graphical window 73 along with a 3D virtual model of the teeth 75 to one side. The 2D digital photograph 71 is scaled up or down in size as necessary to as to be approximately the same in scale (size) as the 3D model of the teeth 75. This is accomplished using any suitable icons or mouse action, such as clicking on the 2D photograph and scrolling up or down with the mouse to change the size of the 2D image so that it matches the size of the 3D model.” [0083] “The computer or workstation 10 (FIG. 1) that includes the software for generating the patient model preferably includes interactive treatment planning software that allows the user to simulate various possible treatments for the patient on the workstation and visualize the results of proposed treatments on the user interface by seeing their effect on the visual appearance of the patient, especially their smile. The interactive treatment planning preferably provides suitable tools and icons that allow the user to vary parameters affecting the patient. Such parameters would include parameters that can be changed so as to simulate change in the age of the patient, and parameters that allow the user to adjust the color, texture, position and orientation of the teeth, individually and as a group. The user manipulates the tools for these parameters and thereby generates various virtual patient models with different features and smiles.” [0105]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Sachdeva into the combination of Sutton, Spilker and Stefan, in order to facilitate visualization and evaluation of the anticipated treatment result.
30. With reference to claim 10, Sutton teaches transmitted to the mobile device. (“this depicts an illustrative diagrammatic representation of a machine in the form of a computer system 1100 within which a set of instructions, when executed, may cause the machine to perform any one or more of the methods discussed herein. … The machine may comprise a server computer, a client user computer, a personal computer (PC), a tablet PC, a smart phone, a laptop computer, a desktop computer, a control system, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. It will be understood that a communication device of the subject disclosure includes broadly any electronic device that provides voice, video or data communication. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. The computer system 1100 may include a processor 1102 (e.g., a central processing unit (CPU), a graphics processing unit (GPU, or both), a main memory 1104 and a static memory 1106, which communicate with each other via a bus 1108.” [0097-0099])
The combination of Sutton and Spilker does not explicitly teach at least one of the 3D model of the predicted post treatment condition of the body part or the view of the 3D model of the predicted post treatment condition of the body part. This is what Stefan teaches (“the preoperative candidate may review the forecasted 3-D images, either alone or together with a surgeon, and select the best overall aesthetic forecasted 3-D image that meets the desires of the patient, and the surgical guidelines, given the preoperative candidate's overall body condition. … once a forecasted 3-D image is selected, the selected forecasted 3-D image is converted from a 3-D image to a format that a scanner can read and measure. In various exemplary embodiments, this is accomplished via a data converter. The data converter takes the forecasted 3-D image, which is in 3-D format, such as, for example a VRML format, and converts the forecasted 3-D image into a format that the scanner can interpret.” col. 5, lines 49-62 “the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 are profiled and displayed. It should be appreciated that, as shown in FIG. 4, the preoperative candidate's image 410 and the postoperative breast augmentation patient images 420 have each been converted from the scan image format to a 3-D image format.” col. 7, lines 8-14) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Stefan into the combination of Sutton and Spilker, in order to provide the patients and potential patients with accurate forecasting and modeling of expected results and outcomes.
31. Claim 18 is similar in scope to the combination of claims 9 and 10, and thus is rejected under similar rationale.
Conclusion
32. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights