DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 82 is objected to because of the following informalities: claim 82 recites generating a predicted 3D model corresponding to an altered representation of the dental site by modifying the 3D model however the modified 3D model is not entirely conclusive whether it is in regard to the 3D model representative of the dental site or of the 3D model representative of the head. Examiner understands the claim to refer the 3D model that is modified as the 3D model of the head, but notes additional clarification as potentially helpful. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 51 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Achraf Ben-Hamadou et al. (Pat. Pub. US-20220175491-A1, herein after “Hamadou”).
In regard to claim 51, Hamadou teaches [a] computer-implemented method comprising:
receiving an image or sequence of images comprising a face of an individual that is representative of a current condition of a dental site of the individual “a first step of fitting the shape of at least one portion of the patients face onto a parametric face shape model based on at least one captured image of the face” (Hamadou, ¶ [0019]), and “a step of capturing, by a capture device, an image of the dentition of the patient” (Hamadou, ¶ [0021]) where images of a face and dentition are taught;
estimating tooth shape of the dental site from the image or sequence of images to generate a 3D model representative of the dental site “a step of modeling the 3D shape of at least one tooth based on a probable position of at least one contour” (Hamadou, ¶ [0023]) where teeth may be modeled, which includes estimating a shape;
generating a predicted 3D model corresponding to an altered representation of the dental site “a step of assembling the reconstructed face shape and the reconstructed dentition in a common 3D virtual space” (Hamadou, ¶ [0024]) where a reconstructed dentition is read as an altered representation of the dental site; and
modifying the image or sequence of images by rendering the dental site to appear as the altered representation based on the predicted 3D model “a step of computing an image of how the patients face will look after the dental treatment based on an image of the patients face captured during the second capture step and on the selected treatment plan” (Hamadou, ¶ [0031]) where the image with the dental treatment is read as the modified image.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 52, 56-58, 81-82, 85, & 119-122 are rejected under 35 U.S.C. 103 as being unpatentable over Hamadou in view of Dmitry Chekh et al. (Pat. Pub. US-20210393375-A1, herein after “Chekh”).
In regard to claim 52, Hamadou teaches [t]he method of claim 51, further comprising:
receiving an initial 3D model representative of the individual's teeth “parametric modeling of the shape of the teeth and modeling of the dentition, in a second landmark, as shown in FIG. 5” (Hamadou, ¶ [0075]).
Hamadou teaches of modeling the teeth with respect to an upper and lower set, but does not explicitly recite that the 3D model [is] corresponding to the upper jaw, the lower jaw, or both.
Chekh teaches that the 3D model [is] corresponding to the upper jaw, the lower jaw, or both “FIG. 43 illustrates a 3D bite model of the upper jaw from both the anterior position on the left and from the occlusal place, on the right” (Chekh, ¶ [0346]).
PNG
media_image1.png
427
680
media_image1.png
Greyscale
Chekh, Fig. 43, depicting a 3D model of the jaw from different views.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the method of a jaw representation taught by Chekh to include the jaw in the model. The motivation to do so would be to have a more comprehensive dental model which is capable of addressing more treatments.
In regard to claim 56, Hamadou in view of Chekh teaches [t]he method of claim 52, further comprising:
segmenting the image or sequence of images to identify teeth within the image or sequence of images to generate segmentation data “manual segmentation of the teeth by means of the 3D scan of teeth: each tooth is cropped from the other teeth and the gums” (Hamadou, ¶ [0144]) where an image can be segmented based on the identified teeth, wherein the segmentation data is representative of shape and position of each identified tooth “the production of this model can be based on a step of statistically learning the shape model of separate teeth: once the database has a large enough number of annotated 3D teeth, the statistical shapes can be leaned for each type of tooth” (Hamadou, ¶ [0148]) where annotations are made based on the segmented teeth and are read as segmentation data. Additionally, “all segmented teeth in three dimensions of the same type are placed in the same reference space” (Hamadou, ¶ [0148]) where position and shape are known, and may be included in the annotated 3D teeth which is read as segmentation data.
In regard to claim 57, Hamadou in view of Chekh teaches [t]he method of claim 56, further comprising:
fitting the 3D model to the image or sequence of images based on the segmentation data by applying a non-rigid fitting algorithm “Variations in 3D shapes are taken into account in the following step, which consists of deforming the generic tooth models to fit as closely as possible the 3D tooth shapes” (Hamadou, ¶ [0148]) where the tooth models are deformed to fit the tooth shape, and the deformation is read as non-rigid fitting.
In regard to claim 58, Hamadou in view of Chekh teaches [t]he method of claim 57, wherein the non-rigid fitting algorithm comprises contour-based optimization to fit the teeth of the 3D model to the teeth identified in the segmentation data “This consists of a deformable adjustment which moves the surface of the generic models to bring them as close as possible to the actual surfaces of the segmented teeth under a certain geometric constraint” (Hamadou, ¶ [0148]) where the deforming includes adjusting the surface to match the segmented teeth. The surface matching is read as contour-based optimization.
In regard to claim 81, Hamadou teaches [a] computer-implemented method comprising:
receiving a video comprising a face of an individual that is representative of a current condition of a dental site of the individual “a step of capturing, by a capture device, an image of the dentition of the patient” (Hamadou, ¶ [0021]), additionally “In some variants, the capture device only captures photographs. In some variants, the capture device only captures videos. In some variants, the capture device captures a combination of photographs and videos” (Hamadou, ¶ [0135]) where imaging is done of the dentition of a patient, which is read as the dental site of the individual, and the imaging may be a video from a capturing device; and
estimating tooth shape of the dental site from the video, wherein the 3D model comprises a representation of the dental site based on the tooth shape estimation “a step of modeling the 3D shape of at least one tooth based on a probable position of at least one contour” (Hamadou, ¶ [0023]) where teeth may be modeled, which includes estimating a shape.
Hamadou teaches of a 3D model of the face and mouth area, but does not teach of generating a 3D model representative of the head of the individual.
Chekh teaches of generating a 3D model representative of the head of the individual “the treatment plan and proposed final or target position of the patient's teeth may be determined with the aid of the facial image of the patient, thereby creating a treatment plan that positions the teeth based on a final orthodontic position that corrects malocclusions and takes into account facial features, such as head shape, eye position, facial midline, and other facial features described herein” (Chekh, ¶ [0199]), additionally, “FIG. 23 depicts a method determining face and head shape of a patient” (Chekh, ¶ [0102]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of videoing a face and 3D modeling of a dental area taught by Hamadou, with the modeling of the head taught by Chekh to model the entire head in 3D based on a video. The motivation to do so would be to have a comprehensive model of the individual’s head that may be obtained from a convenient recording medium.
In regard to claim 82, Hamadou in view of Chekh teaches [t]he method of claim 81, further comprising:
generating a predicted 3D model corresponding to an altered representation of the dental site by modifying the 3D model to alter the representation of the dental site “Each treatment plan has an initial state, corresponding to the state of the model of the patient's dentition, and a final state, corresponding to a secondary model in which at least one tooth has undergone treatment, thereby changing the model of both the dentition and the face shape” (Hamadou, ¶ [0171]) where the predicted 3D model of the dental site is from a selected treatment plan, and shows a modified version of the 3D model.
In regard to claim 85, Hamadou in view of Chekh teaches [t]he method of claim 81, further comprising:
segmenting one or more of a plurality of frames of the video to detect teeth of the individual's dental site “manual segmentation of the teeth by means of the 3D scan of teeth: each tooth is cropped from the other teeth and the gums” (Hamadou, ¶ [0144]) where segmenting can be done on an image, video, or scan, wherein estimating tooth shape comprises applying a non-rigid fitting algorithm “Variations in 3D shapes are taken into account in the following step, which consists of deforming the generic tooth models to fit as closely as possible the 3D tooth shapes” (Hamadou, ¶ [0148]) where the tooth models are deformed to fit the tooth shape, and the deformation is read as non-rigid fitting comprising contour-based optimization to fit the teeth of the 3D model to the teeth identified in the segmentation “This consists of a deformable adjustment which moves the surface of the generic models to bring them as close as possible to the actual surfaces of the segmented teeth under a certain geometric constraint” (Hamadou, ¶ [0148]) where the deforming includes adjusting the surface to match the segmented teeth. The surface matching is read as contour-based optimization.
In regard to claim 119, Hamadou teaches the method of claim 51.
Hamadou fails to teach [a] system comprising:
a memory; and
a processing device operatively coupled to the memory.
Chekh teaches [a] system comprising:
a memory; and
a processing device operatively coupled to the memory “Computer-readable media, computer systems having processors and memory, and computer-implemented methods of orthodontically treating a patient may comprise” (Chekh, ¶ [0010]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the use of a memory and processor taught by Chekh. The motivation to do so would be to include the required computing components to perform the imaging and modeling.
In regard to claim 120, Hamadou teaches the method of claim 51.
Hamadou fails to teach [a] non-transitory machine-readable medium having instructions encoded thereon.
Chekh teaches [a] non-transitory machine-readable medium having instructions encoded thereon “Memory subsystem 408 typically includes a number of memories (e.g., RAM 410, ROM 412, etc.) including computer readable memory for storage of fixed instructions” (Chekh, ¶ [0191]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the use of instructions stored on a memory taught by Chekh. The motivation to do so would be to enable the computer implemented program operate.
In regard to claim 121, Hamadou teaches the method of claim 81.
Hamadou fails to teach [a] system comprising:
a memory; and
a processing device operatively coupled to the memory.
Chekh teaches [a] system comprising:
a memory; and
a processing device operatively coupled to the memory “Computer-readable media, computer systems having processors and memory, and computer-implemented methods of orthodontically treating a patient may comprise” (Chekh, ¶ [0010]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the use of a memory and processor taught by Chekh. The motivation to do so would be to include the required computing components to perform the imaging and modeling.
In regard to claim 122, Hamadou teaches the method of claim 81.
Hamadou fails to teach [a] non-transitory machine-readable medium having instructions encoded thereon.
Chekh teaches [a] non-transitory machine-readable medium having instructions encoded thereon “Memory subsystem 408 typically includes a number of memories (e.g., RAM 410, ROM 412, etc.) including computer readable memory for storage of fixed instructions” (Chekh, ¶ [0191]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the use of instructions stored on a memory taught by Chekh. The motivation to do so would be to enable the computer implemented program operate.
Claims 53-55 & 83-84 are rejected under 35 U.S.C. 103 as being unpatentable over Hamadou in view of Chekh and Ryan Amelon et al. (Pat. Pub. US-20230363857-A1, herein after “Amelon”).
In regard to claim 53, Hamadou in view of Chekh teaches [t]he method of claim 52, further comprising:
encoding the initial 3D model via a trained machine learning model “The patient's facial landmarks within the image are computed using a machine learning algorithm, such as a convoluted neural network” (Chekh, ¶ [0230]) where machine learning is used.
Hamadou in view of Chekh fail to teach encoding the initial 3D model into a latent space vector.
Amelon teaches encoding the initial 3D model into a latent space vector “By using previous or historic treatment plan data to train the machine learning models set forth herein, the systems and methods described herein may learn to leverage full and actual 3D geometries for training or learning treatment planning movements” (Amelon, ¶ [0042]), and “encoding the compressed tooth representation into a latent space representation (e.g., vector)” (Amelon, ¶ [0075]) where a machine learning process can encode a model into a latent space vector.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou in view of Chekh, with the use of a latent space vector taught by Amelon to transform the model into a vector representation. The motivation to do so would be to improve the machine learning aspect.
In regard to claim 54, Hamadou in view of Chekh and Amelon teaches [t]he method of claim 53, wherein the trained machine learning model is a variational autoencoder “if the geometric encoder model 1006 is an autoencoder, an encoder portion of the autoencoder may learn the latent space representation of one or more teeth in the teeth representation 1004 (e.g., compressing the tooth” (Amelon, ¶ [0076]). The motivation to combine would be the same as claim 53.
In regard to claim 55, Hamadou in view of Chekh and Amelon teaches [t]he method of claim 53, wherein the trained machine learning model is trained to predict post-treatment modification of the initial 3D model and generate the predicted 3D model from the predicted post-treatment modification “Referring to FIG. 11, a block diagram of an example system 1100 using supervised learning that may be used to determine a movement of the teeth (e.g., the final position of one or more of the patient's teeth post treatment described with respect to the final tooth orientation and translation) is shown according to an example embodiment” (Amelon, ¶ [0082]) where the model may generate what the patient’s teeth look like after being moved as a treatment. The motivation to combine would be the same as claim 53.
In regard to claim 83, Hamadou in view of Chekh teaches [t]he method of claim 82.
Hamadou in view of Chekh fail to teach encoding the 3D model into a latent space vector via a trained machine learning model, wherein the trained machine learning model is a variational autoencoder.
Amelon teaches encoding the 3D model into a latent space vector via a trained machine learning model “By using previous or historic treatment plan data to train the machine learning models set forth herein, the systems and methods described herein may learn to leverage full and actual 3D geometries for training or learning treatment planning movements” (Amelon, ¶ [0042]), and “encoding the compressed tooth representation into a latent space representation (e.g., vector)” (Amelon, ¶ [0075]) where a machine learning process can encode a model into a latent space vector, wherein the trained machine learning model is a variational autoencoder “if the geometric encoder model 1006 is an autoencoder, an encoder portion of the autoencoder may learn the latent space representation of one or more teeth in the teeth representation 1004 (e.g., compressing the tooth” (Amelon, ¶ [0076]). The motivation to combine would be the same as claim 53.
In regard to claim 84, Hamadou in view of Chekh and Amelon teaches [t]he method of claim 83, wherein the trained machine learning model is trained to predict post-treatment modification of the 3D model and generate the predicted 3D model from the predicted post-treatment modification “Referring to FIG. 11, a block diagram of an example system 1100 using supervised learning that may be used to determine a movement of the teeth (e.g., the final position of one or more of the patient's teeth post treatment described with respect to the final tooth orientation and translation) is shown according to an example embodiment” (Amelon, ¶ [0082]) where the model may generate what the patient’s teeth look like after being moved as a treatment. The motivation to combine would be the same as claim 53.
Claims 86-87 are rejected under 35 U.S.C. 103 as being unpatentable over Hamadou in view of Chekh and Xavier Artizzu et al. (Pat. Pub. US-20230260238-A1, herein after “Artizzu”).
In regard to claim 86, Hamadou in view of Chekh teaches [t]he method of claim 82.
Hamadou in view of Chekh fail to teach generating a video comprising renderings of the predicted 3D model.
Artizzu teaches generating a video comprising renderings of the predicted 3D model “Once the 3D head model (2) has been animated to reflect the action represented in the 2D images (5), the 3D denture model (4) needs to be moved correspondingly, taking into account the movement of the mandible, lips, and maxillary” (Artizzu, ¶ [0079]) where the head model is animated and the mouth model is animated to match.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou in view of Chekh, with the use of an animation of the 3D model taught by Artizzu to allow the individual to view the results. The motivation to do so would be to let the patient see the results before continuing with their treatment.
In regard to claim 87, Hamadou in view of Chekh teaches [t]he method of claim 81.
Hamadou in view of Chekh fail to teach generating a video comprising renderings of the 3D model.
Artizzu teaches generating a video comprising renderings of the 3D model “an animation of the 3D head model (2) is performed based on the 2D input images” (Artizzu, ¶ [0075]) where the head model is animated based on a provided sequence of 2D images. The motivation to combine would be the same as claim 86.
In regard to claim 88, Hamadou in view of Chekh and Amelon teaches [t]he method of claim 84.
Hamadou in view of Chekh and Amelon fail to explicitly teach receiving a driver sequence comprising a plurality of animation frames, each frame comprising a representation that defines the position, orientation, shape, and expression of a face;
animating the 3D model or the predicted 3D model based on the driver sequence; and
generating a video for display based on the animated 3D model.
Artizzu teaches receiving a driver sequence comprising a plurality of animation frames, each frame comprising a representation that defines the position, orientation, shape, and expression of a face “Inputs are the 2D video (5) and the 3D head model (2). The output is the 3D animated model (6) mimicking the movement observed in the 2D images (5), both in terms of the general head position as well as its facial physiognomy” (Artizzu, ¶ [0076]) where a sequence of 2D frames are input to represent the desired animation including facial expression, and the 3D model is both generated and animated based on the observed head in the provided frames, this includes position and head movements ¶ [0073];
animating the 3D model or the predicted 3D model based on the driver sequence “an animation of the 3D head model (2) is performed based on the 2D input images” (Artizzu, ¶ [0075]) where the head model is animated based on a provided sequence of 2D images; and
generating a video for display based on the animated 3D model “Finally, as shown in FIG. 2, once the head and denture 3D models (2,4) are in their correct position, the final 3D scene rendering (7) is computed and projected back to 2D, obtaining a simulation video (8)” (Artizzu, ¶ [0081]) where a simulated video is output based on the animated model. The motivation to combine would be the same as claim 86.
Claims 91-92, 94-98 & 123-124 are rejected under 35 U.S.C. 103 as being unpatentable over Hamadou in view of Islam Raslambekov et al. (Pat. US-11833007-B1, herein after “Raslambekov”).
In regard to claim 91, Hamadou teaches [a] method comprising:
obtaining, by a processing device, video data of a dental patient comprising a plurality of frames “a first step of capturing, by an RGB-D capture device, at least one image of the patients face” (Hamadou, ¶ [0018]) where multiple frames, or a video, can be captured;
obtaining an indication of first selection criteria in association with the video data, wherein the first selection criteria comprise one or more conditions related to a target dental treatment of the dental patient “step 150 of selecting a treatment plan from the set of determined treatment plans” (Hamadou, ¶ [0098]) where a dental treatment plan may be selected.
Hamadou fails to teach performing an analysis procedure on the video data, wherein performing the analysis procedure comprises:
determining a respective first score for each of the plurality of frames based on the first selection criteria, and
determining that a first frame of the plurality of frames satisfies a first threshold condition based on the first score; and
selecting the first frame responsive to determining that the first frame satisfies the first threshold condition.
Raslambekov teaches performing an analysis procedure on the video data, wherein performing the analysis procedure comprises:
determining a respective first score for each of the plurality of frames based on the first selection criteria “the present methods and systems are directed to determining, during the course of a given orthodontic treatment plan, if there is a deviation value between actual and planned positions of at least one tooth of a subject's teeth” (Raslambekov, Col. 10, Lines 30-35) where a treatment plan is made based on a provided video or scan of the teeth, which is read as a plurality of frames, and the deviation value is read as a score, and
determining that a first frame of the plurality of frames satisfies a first threshold condition based on the first score “a respective actual position 802 of the given tooth 17 can be different from a respective planned position 804 of the given tooth 17, with a deviation value 806. In other words, in the example of FIG. 8, the given tooth 17 is falling behind the given orthodontic treatment plan as, by the given treatment step 714, it has not reached the respective planned position 804 associated with the given treatment step 714” (Raslambekov, Col. 15, Lines 16-25) where a provided tooth image may or may not be at a planned position, and “modelling the respective planned positions of the upper teeth 16 at the given treatment step 714 of the current orthodontic treatment plan in the initial 3D digital model” (Raslambekov, Col. 24, Lines 1-5) where the upper teeth may be modeled with a set of frames; and
selecting the first frame responsive to determining that the first frame satisfies the first threshold condition “the processor 950 can be configured to visualize, for example, on the planned 3D digital model 1202, the deviation values determined for each tooth of the upper teeth 16, for example, in a form of a heat map representation 1602, schematically depicted in FIG. 16, in accordance with certain non-limiting embodiments of the present technology” (Raslambekov, Col. 31, Lines 51-55) where the images of each tooth, or tooth frames are selected to display each representation of their deviation value via a predetermined color, which is read as selecting the first frame as no specific detail or step is provided beyond selecting the frame responsive to the frame satisfying the threshold condition.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a face and modeling a dental area taught by Hamadou, with the criteria and frame scoring taught by Raslambekov to allow a patient or dentist to select a treatment plan and compare the current model to the treatment plan images. The motivation to do so would be to see the progression of the treatment and visualize the current and next steps.
In regard to claim 92, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the analysis procedure further comprises:
determining that a second frame of the plurality of frames satisfies a first criterion of the first selection criteria “a respective actual position 802 of the given tooth 17 can be different from a respective planned position 804 of the given tooth 17, with a deviation value 806. In other words, in the example of FIG. 8, the given tooth 17 is falling behind the given orthodontic treatment plan as, by the given treatment step 714, it has not reached the respective planned position 804 associated with the given treatment step 714” (Raslambekov, Col. 15, Lines 16-25) where a provided tooth image may or may not be at a planned position, and “modelling the respective planned positions of the upper teeth 16 at the given treatment step 714 of the current orthodontic treatment plan in the initial 3D digital model” (Raslambekov, Col. 24, Lines 1-5) where the upper teeth may be modeled with a set of frames and each frame may have the same determining process performed;
determining that a third frame of the plurality of frames satisfies a second criterion of the first selection criteria “Orthodontic treatments are used for treating different conditions relating to teeth misalignment or malocclusion, including but not limited to one or more of: tooth rotation, tooth intrusion/extrusion, tooth translation, and interdental space management” (Raslambekov, Col. 10, Lines 58-61) where several different treatments may be addressed in criteria selection on positioning of teeth, and multiple frames can be used to model the teeth of a patient; and
PNG
media_image2.png
358
482
media_image2.png
Greyscale
Raslambekov, Fig. 16, depicting the upper row of teeth (item 1202) with each individual tooth (item 17) has a calculated deviation value (item 1602) based on a selection criteria. When imaging the teeth each tooth may be read as its own frame, where a tooth may have a high deviation value (item 806) or low deviation value.
generating the first frame based on a portion of the second frame associated with the first criterion and a portion of the third frame associated with the second criterion “an actual 3D digital model including a first plurality of points representative of surfaces of the subject's teeth in actual positions thereof at the given stage of the current orthodontic treatment plan” (Raslambekov, Col. 7, Lines 31-34) where the tooth frames and their deviation values can be combined to make the model.
In regard to claim 94, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the analysis procedure further comprises:
generating, based on the video data, a three-dimensional model of the dental patient “a second step of fitting the patients face shape onto a parametric face shape model based on at least one image of the face captured during the second capture step, in a 3D space” (Hamadou, ¶ [0027]) where the face model can be obtained from a video; and
rendering the first frame based on the three-dimensional model “during the step 110 of capturing at least one image of the patients face shape, at least one captured image is in two dimensions. These embodiments make the step 120 of fitting the face shape in three dimensions more complex, but make it possible to use capture devices that are less expensive and more widely available” (Hamadou, ¶ [0108]) where a first captured image, or first frame is used for modeling the patient’s face, the face model can then be viewed based on the captured image.
In regard to claim 95, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the indication of the first selection criteria comprises a reference image, wherein a score of the reference image in association with the first selection criteria satisfies the first threshold condition “instead of using a single reference point, that is, the reference point 1302, the processor 950 can be configured to use a plurality of reference points, including, for example, two, three, ten, or twenty reference points. Similarly, in these embodiments, the processor 950 can be configured to: (i) identify each one of the plurality of reference points along or within the surface of the given tooth 17 in each one of the actual and planned 3D digital models 1102, 1202; and (ii) merge respective instances of each one of the plurality of reference points as described above with respect to the reference point 1302” (Raslambekov, Cols. 26-27) where a planned model can be drawn to show what a tooth orientation with no deviation would look like based on a reference point.
PNG
media_image3.png
598
527
media_image3.png
Greyscale
Raslambekov, Fig. 13, depicting a reference point (item 1302) utilized to align a tooth (item 1102) with a planned model (item 1202) that has no deviation.
In regard to claim 96, Hamadou in view of Raslambekov teaches [t]he method of claim 91, further comprising:
obtaining an indication of second selection criteria “Orthodontic treatments are used for treating different conditions relating to teeth misalignment or malocclusion, including but not limited to one or more of: tooth rotation, tooth intrusion/extrusion, tooth translation, and interdental space management” (Raslambekov, Col. 10, Lines 58-61) where several different treatments may be addressed in criteria selection on positioning of teeth;
wherein the analysis procedure further comprises:
determining a respective second score for each of the plurality of frames based on the second selection criteria “the present methods and systems are directed to determining, during the course of a given orthodontic treatment plan, if there is a deviation value between actual and planned positions of at least one tooth of a subject's teeth” (Raslambekov, Col. 10, Lines 30-35) where a score can be provided based on a deviation for any selected misalignment or malocclusion, a dentist or professional could image a patient’s teeth using multiple frames and score the frames based on a number of selected criteria; and
determining that a second frame satisfies a second threshold condition based on the second score “the imaging device 930 may also be configured to capture and/or process image data of the lower arch form (not depicted) of the subject without departing from the scope of the present technology. It should also be noted that the image data may include two-dimensional (2D) data and/or three-dimensional data (3D)” (Raslambekov, Col. 17, Lines 47-51) where the lower arch is imaged, which can be read as a second frame, alternatively, the plurality of images taken to generate a 3D model of the patient’s dental arches can be read as including a second frame. Furthermore, the processing performed on the first frame, or upper arch, are repeated or are able to be replicated to the lower arch, such as satisfying a score, or deviation value; and
selecting the second frame responsive to determining that the second frame satisfies the second threshold condition “the processor 950 can be configured to visualize, for example, on the planned 3D digital model 1202, the deviation values determined for each tooth of the upper teeth 16, for example, in a form of a heat map representation 1602, schematically depicted in FIG. 16, in accordance with certain non-limiting embodiments of the present technology” (Raslambekov, Col. 31, Lines 51-55) where the images of each tooth, or tooth frames are selected to display each representation of their deviation value via a predetermined color, which is read as selecting the second frame since either the next tooth in the upper row can be read as a second frame, or the act of repeating the display on the bottom row can be read as a second frame and no specific detail or step is provided beyond selecting the frame responsive to the frame satisfying the threshold condition. The motivation to combine Raslambekov is the same as claim 91.
In regard to claim 97, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the first selection criteria comprise values associated with one or more of:
head orientation;
visible tooth identities;
visible tooth area;
bite position;
emotional expression; or
gaze direction “… the deviation value 806 is minimized by an end of the given orthodontic treatment plan. More specifically, the present methods and systems are directed to: (i) aligning 3D digital models associated with the upper teeth 16 that are representative of the planned and actual positions of the upper teeth 16 at the given treatment step 714 of the given orthodontic treatment plan; (ii) determining the deviation value 806; and (iii) adjusting the respective tooth trajectory 708 so as to minimize the deviation value 806 by the end of the given orthodontic treatment” (Raslambekov Col. 15, Lines 49-58) where the goal of aligning the teeth and tooth trajectory is read as bite position, tooth identities, or visible tooth area.
In regard to claim 98, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the video data comprises a first portion obtained at a first time and a second portion obtained at a second time, the second portion comprising the first frame “It should be noted that although the examples provided herein are directed to receiving and processing the image data of the upper arch form 24 of the subject, it is done solely for the purposes of clarity of explanation of certain non-limiting embodiments of the present technology; therefore, the imaging device 930 may also be configured to capture and/or process image data of the lower arch form” (Raslambekov, Col. 17, Lines 62-67) where video data is a scan of the arches of a patient and a first and second portion may be the upper and lower arches scanned at different times, and wherein the analysis procedure further comprises:
determining that scores associated with each of the frames of the first portion do not satisfy the first threshold “the graphical user interface 750 may include the planned schedule 700 and a 3D digital model 770 representing planned tooth positions of the upper teeth 16 and the lower teeth 27 at each one of the plurality of treatment steps 706 in the course of the orthodontic treatment” (Raslambekov, Col. 15, Lines 5-10), and “to determine and output, based on image data associated with the subject's teeth, the deviation values between the actual and planned positions of the subject's teeth at the given treatment step 714 of the given orthodontic treatment plan” (Raslambekov, Col. 16, Lines 36-40) where the deviation value can be used to determine the scores and each frame of each portion, or each tooth may be scored as above or below the threshold; and
providing an alert to a user indicating one or more criteria of the first selection criteria to be included in the second portion “the planned schedule 700 may be displayed as a graphical user interface 750 (for example, on a screen 922 of the system 900 schematically depicted in FIG. 9). The planned schedule 700 may thus assist an orthodontic clinician (such as an orthodontist, for example) to monitor the so planned orthodontic treatment and allow modification thereof” (Raslambekov, Cols. 14-15) where the user is shown the treatment process throughout for each portion.
In regard to claim 123, Hamadou in view of Raslambekov teaches [a] system comprising:
a memory; and
a processing device operatively coupled to the memory, wherein the processing device is configured to perform the method of claim 91 “non-limiting embodiments of the present technology, the processor 950 may be configured to store the heat map representation 1070 in one of the solid-state drive 960 and the random-access memory 970” (Raslambekov, Col. 32, Lines 5-10).
In regard to claim 124, Hamadou teaches [a] non-transitory machine-readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to perform the method of claim 91 “The electronic device comprises a non-transitory computer-readable medium storing instructions; and a processor, which, upon executing the instructions, is configured to …” (Raslambekov, Col. 7, Lines 30-35).
Claim 93 is rejected under 35 U.S.C. 103 as being unpatentable over Hamadou in view of Raslambekov and Amelon.
In regard to claim 93, Hamadou in view of Raslambekov teaches [t]he method of claim 91, wherein the analysis procedure further comprises:
determining that a second frame of the plurality of frames satisfies a first criterion of the first selection criteria “a respective actual position 802 of the given tooth 17 can be different from a respective planned position 804 of the given tooth 17, with a deviation value 806. In other words, in the example of FIG. 8, the given tooth 17 is falling behind the given orthodontic treatment plan as, by the given treatment step 714, it has not reached the respective planned position 804 associated with the given treatment step 714” (Raslambekov, Col. 15, Lines 16-25) where a provided tooth image may or may not be at a planned position, and “modelling the respective planned positions of the upper teeth 16 at the given treatment step 714 of the current orthodontic treatment plan in the initial 3D digital model” (Raslambekov, Col. 24, Lines 1-5) where the upper teeth may be modeled with a set of frames and each frame may have the same determining process performed;
determining that the second frame does not satisfy a second criterion of the first selection criteria “Orthodontic treatments are used for treating different conditions relating to teeth misalignment or malocclusion, including but not limited to one or more of: tooth rotation, tooth intrusion/extrusion, tooth translation, and interdental space management” (Raslambekov, Col. 10, Lines 58-61) where several different treatments may be addressed in criteria selection on positioning of teeth, and
Hamadou in view of Raslambekov fail to explicitly teach providing the second frame to a trained machine learning model; and
obtaining the first frame from the trained machine learning model, wherein the first frame is based on the second frame, satisfies the first criterion, and satisfies the second criterion.
Amelon teaches providing the second frame to a trained machine learning model “supervised learning that may be used to determine a movement of the teeth (e.g., the final position of one or more of the patient's teeth post treatment described with respect to the final tooth orientation and translation)” (Amelon, ¶ [0082]) where an image of the current teeth can be provided to a machine learning model; and
obtaining the first frame from the trained machine learning model, wherein the first frame is based on the second frame, satisfies the first criterion, and satisfies the second criterion “The final position model 1010 may be trained on known input-output pairs (e.g., full 3D geometric encoded and compressed representations of initial teeth positions and final teeth positions) such that the final position model 1010 can learn how to predict known outputs given known inputs. Once a final position model 1010 has learned how to predict known input-output pairs, the final position model 1010 can operate on unknown inputs to predict an output” (Amelon, ¶ [0082]) where the input image is given to the machine learning model to make a corresponding output image that satisfies a given criteria “the final position of the previous patient's dentition (which may be rotation components and translation components obtained from the treatment plan” (Amelon, ¶ [0084]) where translation and rotation can be read as the first and second criterion.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of imaging a patient’s face with teeth and modeling a dental area taught by Hamadou in view of Raslambekov, with the use of a machine learning model taught by Amelon to transform the frame provided into an improved model. The motivation to do so would be to produce a patient expected image that satisfies the selected criteria.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Dmitry Mednikov et al. (Pat. US-11553988-B2) discloses 3D modeling of a patient’s dentition using reference points to generate modified images (abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights