DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-14,16-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kopelman (PGPUB: 20160256035) in view of Dawood (PGPUB: 20200170760), and further in view of Jo (PGPUB: 20200008877 A1).
Regarding claims 1, 10, and 20. Kopelman teaches a system comprising:
an intraoral scanner configured to generate image data of a dental site (see Fig. 2-6, paragraph 65, an intraoral scan session is started. During the intraoral scan session, a dental practitioner uses an intraoral scanner to create a set of intraoral images focused on a particular intraoral site); and
a computing device configured to:
receive the image data (see Fig. 2, paragraph 65, processing logic receives a set of intraoral images of the intraoral site);
analyze the image data to identify a captured surface on a reference object at the dental site (see paragraph 83, Processing logic may provide an indication of the anomaly via a user interface. The contour of the anomaly may be displayed in manner to contrast the anomaly from surrounding imagery. For example, teeth may be shown in white, while the anomaly may be shown in red, black, blue, green, or another color), wherein the reference object is a foreign object intentionally added to the dental site to facilitate imaging of the dental site (see paragraph 46, Processing logic may provide an indication of the anomaly via a user interface. The contour of the anomaly may be displayed in manner to contrast the anomaly from surrounding imagery. For example, teeth may be shown in white, while the anomaly may be shown in red, black, blue, green, or another color);
identify the reference object and determine a position and orientation of the reference object at the dental site based at least in part on the captured surface pattern on the reference object (see Fig, 7, paragraph 67, Each of the intraoral images 714-718 may have been generated by an intraoral scanner having a particular distance from the dental surface being imaged. At the particular distance, the intraoral images 714-718 have a particular scan area and scan depth. The shape and size of the scan area will generally depend on the scanner, and is herein represented by a rectangle. Each image may have its own reference coordinate system and origin. Each intraoral image may be generated by a scanner at a particular position ( scanning station). The location and orientation of scanning stations may be selected such that together the intraoral images adequately cover an entire target zone); and
generate a three-dimensional (3D) model of the dental site based at least in part on the image data (see paragraph 70, processing logic generates a virtual 3D model that includes the intraoral site. The selected portions of the locked intraoral image (e.g., that are inside of the determined contour) are used to create a first region of the model . For example, the selected portions may be used to create a particular preparation tooth in the 3D model. Data from the additional intraoral images are not used to create the region of the 3D model).
However, Kopelman does not expressly teach surface pattern.
Dawood teaches that the processor 22 receives 3D point cloud data from the metrology system 14 and colour image data from the imaging system 18. The combined data of 3D point cloud data and the corresponding colour image as acquired for a surface of a given part of an intraoral scene is herein referred to as a scanning dataset. The colour image of a scanning dataset is processed to label the image elements that are within a region of the image representing a surface of said intraoral scene , which should preferably not be included in said 3D representation. Typically, image elements are labelled within a region with either a colour or colour pattern corresponding to a surface colour or surface colour pattern of a utensil used intraorally when acquiring said scanning dataset. Alternatively, or in addition, image elements are labelled that are within a region having a colour pattern corresponding to a colour pattern of a tooth surface area comprising undesired stains or particles (see paragraph 80).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kopelman by Dawood to obtain The colour image of a scanning dataset is processed to label the image elements that are within a region of the image representing a surface of said intraoral scene , which should preferably not be included in said 3D representation. Typically, image elements are labelled within a region with either a colour or colour pattern corresponding to a surface colour or surface colour pattern of a utensil used intraorally when acquiring said scanning dataset, in order to provide surface pattern. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
However, the combination does not expressly teach wherein the identified reference object is used to augment the 3D model of the dental site.
Jo teaches that the system may further comprise an X-ray Scanner 28 (CBCT/MRI Scanner) for obtaining preoperative CBCT/MRI scans of a treatment site, an intra oral scanner 30 for obtaining preoperative 3D images of the patient's mouth and/or a 3D Face Scanner 36 for obtaining a 3D scan of the face. A camera system 3 such as a 3D optical tracking system and/or stereoscopic camera system may be included in the computer system and may form or be a part of the tracking means 2. Alternatively, the camera system 3 may be embedded in the display device 12 of the clinician 10 or may be a part of the tracking means 2 (see Fig. 1, paragraph 22); The processor 122 may be configured to receive CTCB/MRI data 18, intraoral images 20 and/or facial scan and geometrically register them together to be overlaid onto the patient 14 through for example see-through Augmented Reality Glasses/HUD display or onto a stereoscopic video of the patient using e.g. through a head mounted stereoscopic display. A treatment plan, including a planned implant location 32 (the planned implant location optionally including a planned/desired drill depth) may also be geometrically registered with the CBCT/MRI 18 and intra-oral 20 images and/or 2D/3D facial scan and overlaid onto the treatment site 14a (see Fig. 1, paragraph 23).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Jo to obtain that the processor 122 may be configured to receive CTCB/MRI data 18, intraoral images 20 and/or facial scan and geometrically register them together to be overlaid onto the patient 14 through for example see-through Augmented Reality Glasses/HUD display or onto a stereoscopic video of the patient using e.g. through a head mounted stereoscopic display. A treatment plan, including a planned implant location 32 (the planned implant location optionally including a planned/desired drill depth) may also be geometrically registered with the CBCT/MRI 18 and intra-oral 20 images and/or 2D/3D facial scan and overlaid onto the treatment site 14a, in order to provide wherein the identified reference object is used to augment the 3D model of the dental site. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 2 and 11. The combination teaches the system of claim 10, wherein identifying the reference object comprises:
comparing the captured surface pattern on the reference object to known surface patterns of a plurality of reference objects (see Dawood, paragraph 40-44, the determining whether said colour pattern of said pattern region matches said utensil surface colour pattern comprises: calculating two or more combined colour surface areas by adding surface areas of the respective colour regions in the pattern region, which comprise image elements having a colour code within a same range, determining a ratio of the combined colour surface areas); and
determining a match between the captured surface pattern and a known surface pattern of the reference object (see Kopelman, Fig. 2, paragraph 68, the selected portions may correspond to a contour of a tooth or other feature of the intraoral site. The selected portions may be determined based on performing image analysis and applying object recognition techniques, such as edge detection, edge matching, greyscale matching, gradient matching, bag of words models, and so on. Reference data may be used to train processing logic to detect particular objects such as teeth. In one embodiment, the known identity of the tooth or intraoral site is used to assist the object detection process and to select the portions of the intraoral images; see Dawood, paragraph 40, determining whether the colour pattern of the pattern region matches the utensil surface colour pattern comprises analyzing relative positions within the pattern region of the two or more colour regions in relation to relative positions of the one or more corresponding colour areas in said surface colour pattern of said utensil).
Regarding claims 3 and 12. The combination teaches the system of claim 10, wherein the computing device is further configured to:
output the 3D model to a display (see Kopelman, Fig. 7).
Regarding claims 4 and 13. The combination teaches the system of claim 10,
wherein the image data comprises three-dimensional (3D) intraoral scan data (see Kopelman, Fig. 7).
Regarding claims 5 and 14. The combination teaches the system of claim 10,
wherein the computing device is further configured to:
determine a 3D shape of the reference object for the 3D model based on a known 3D shape of the reference object (see Kopelman, paragraph 47, anomaly identifying module 115 to identify anomalies by performing image processing to identify an unexpected shape, a region with low clarity, a region missing data, color discrepancies, and so forth. Different criteria may be used to identify different classes of anomalies. In one embodiment, an area of missing image data is used to identify anomalies that might be voids. For example, voxels at areas that were not captured by the intraoral images may be identified. In one embodiment, anomaly identifying module interpolates a shape for the anomaly based on geometric features surrounding the anomaly and/or based on geometric features of the anomaly (if such features exist). Such geometric features may be determined by using edge detection, corner detection, blob detection, ridge detection, Hough transformations, structure tensors, and/or other image processing techniques); and
update the 3D model based on the known 3D shape of the reference object and the determined position and orientation of the reference object at the dental site (see Kopelman, paragraph 85, processing logic receives an additional image of the intraoral site. The additional image may include data for the region of the 3D model or initial set of intraoral images where the anomaly was detected. At block 520, processing logic updates the virtual 3D model based on replacing the data of the original set of intraoral images within the border or contour with additional data from the additional image of the intraoral site. Thus, the anomaly may be corrected without affecting a remainder of the virtual 3D model).
Regarding claims 7 and 16. The combination teaches the system of claim 10,
wherein the computing device is further configured to:
remove a representation of the reference object from the 3D model (see Kopelman, paragraph 59, via the user interface, a user may mark or otherwise demarcate the unacceptable portion of the 3D model. Eraser module 132 may then delete or otherwise remove the marked portion from the 3D model (and the associated portions of a locked image data set and/or other image data set used to create the unacceptable portion). For example, the dental procedure of interest may be providing a dental prosthesis, and the deleted or removed part of the 3D model may be part of a finish line of a tooth preparation that exists in a real dental surface, but was not clearly represented in the 3D model (or in the intraoral image data sets 135A-135N used to create the 3D model)).17
Regarding claims 8 and 17. The combination teaches the system of claim 10, wherein the reference object is a scan body, an implant abutment, or an attachment (see Kopelman, paragraph 27, a user (e.g., a practitioner) may subject a patient to intraoral scanning. In doing so, the user may apply scanner 150 to one or more patient intraoral locations. The scanning may be divided into one or more segments. As an example, the segments may include a lower buccal region of the patient, a lower lingual region of the patient, a upper buccal region of the patient, an upper lingual region of the patient, one or more preparation teeth of the patient (e.g., teeth of the patient to which a dental device such as a crown or other dental prosthetic will be applied), one or more teeth which are contacts of preparation teeth (e.g., teeth not themselves subject to a dental device but which are located next to one or more such teeth or which interface with one or more such teeth upon mouth closure), and/or patient bite (e.g., scanning performed with closure of the patient's mouth with the scan being directed towards an interface area of the patient's upper and lower teeth)).
Regarding claims 9 and 18. The combination teaches the system of claim 10, wherein unscanned portions of the reference object are added to the 3D model from reference data about the reference object (see Kopelman, Fig. 1, paragraph 27, the user may apply scanner 150 to one or more patient intraoral locations. The scanning may be divided into one or more segments. As an example, the segments may include a lower buccal region of the patient, a lower lingual region of the patient, a upper buccal region of the patient, an upper lingual region of the patient, one or more preparation teeth of the patient (e.g., teeth of the patient to which a dental device such as a crown or other dental prosthetic will be applied), one or more teeth which are contacts of preparation teeth (e.g., teeth not themselves subject to a dental device but which are located next to one or more such teeth or which interface with one or more such teeth upon mouth closure), and/or patient bite (e.g., scanning performed with closure of the patient's mouth with the scan being directed towards an interface area of the patient's upper and lower teeth) . Via such scanner application, the scanner 150 may provide image data (also referred to as scan data) to computing device 105. The image data may be provided in the form of intraoral image data sets 135A-135N, each of which may include 2D intraoral images and/or 3D intraoral images of particular teeth and/or regions of an intraoral site).
Claim(s) 6, 15, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kopelman (PGPUB: 20160256035) in view of Dawood (PGPUB: 20200170760), in view of Jo (PGPUB: 20200008877 A1), and further in view of Cofar (PGPUB: 20220117708).
Regarding claims 6 and 15. The combination teaches the system of claim 10, wherein the computing device is further configured to: receive an indication that the reference object has been intentionally added to the dental site;
receive one or more known physical properties of the reference object, the one or more known physical properties comprising at least one of a known surface pattern of the reference object or a known 3D shape of the reference object (see Kopelman, paragraph 6, an identified void may be a void in a surface of an image. Examples of surface conflict include double incisor edge and/or other physiologically unlikely tooth edge, bite line shift, inclusion or lack of blood, saliva and/or foreign objects, differences in depictions of a margin line, and so on. The anomaly identifying module 115 may, in identifying an anomaly, analyze patient image data (e.g., 3D image point clouds) and/or one or more virtual 3D models of the patient alone and/or relative to reference data 138. The analysis may involve direct analysis (e.g., pixel-based and/or other point-based analysis), the application of machine learning, and/or the application of image recognition. Such reference data 138 may include past data regarding the at-hand patient (e.g., intraoral images and/or virtual 3D models), pooled patient data, and/or pedagogical patient data, some or all of which may be stored in data store 110).
However, the combination does not expressly teach to query a library using an identification of the reference object.
Cofar teaches that to provide a computer implemented method and a computer program product for finding a limited number of virtual teeth in a digital database or digital library matching a tooth in an oral space of a patient, e.g. having a matching score higher than a predefined score, preferably in a fast and efficient manner; to provide a computer implemented method and a computer program product for building a digital database or library of teeth (e.g. of natural teeth) which is searchable in a fast an efficient manner (see paragraph 10-11).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Cofar to obtain number of virtual teeth in a digital database or digital library matching a tooth in an oral space of a patient, e.g. having a matching score higher than a predefined score, preferably in a fast and efficient manner, in order to provide to query a library using an identification of the reference object. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claim 19. The combination does not expressly teach system of claim 10, wherein the computing device is further configured to:
search through library data of a plurality of reference objects in a library to identify that a surface pattern of the reference object in the image data matches a surface pattern of the reference object in the library data for the reference object.
Dawood teaches that ] obtaining a scanning dataset, which comprises 3D point cloud data representing a part of the intraoral scene in a point cloud coordinate space and a colour image of the part of the intraoral scene in a camera coordinate space, [0023] labelling image elements of the colour image within a region having a colour or colour pattern corresponding either to (i) a surface colour or surface colour pattern of a utensil used intraorally while obtaining the scanning dataset or (ii) to a colour pattern corresponding to a colour pattern of a tooth surface area comprising undesired stains or particles, [0024] filtering out of said 3D point cloud data, data points that map to labelled image elements of the colour image, generating a 3D representation from the filtered 3D point cloud data (see paragraph 22-24).
Cofar teaches that according to a seventh aspect, the present invention also provides a computer implemented method of searching in a digital database of teeth as can be generated by a method according to the second, third or fourth aspect, and of automatically retrieving from the database a limited set of candidate teeth, the method comprising the steps of: a) obtaining or determining a first limited set of parameters that characterize the existing tooth or the envisioned tooth, using a method according to the first aspect; b) creating a list or an array of objects, each object containing at least a pointer or a reference to items of the database, and a matching score; For at least a subset of the digital teeth stored in the digital library , performing the steps c) to e), including c) retrieving a second limited set of parameters of the tooth selected from the digital database; d) calculating a matching score based on the first limited set of parameters and the second limited set of parameters; e) updating said list or array so as to keep pointers or references to the candidate teeth having the highest score; f) providing the list or array with pointers or a references to matching teeth, and their matching score (see paragraph 49).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Cofar to obtain searching in a digital database of teeth as can be generated by a method according to the second, third or fourth aspect, and of automatically retrieving from the database a limited set of candidate teeth, the method comprising the steps of: a) obtaining or determining a first limited set of parameters that characterize the existing tooth or the envisioned tooth, using a method according to the first aspect; b) creating a list or an array of objects, each object containing at least a pointer or a reference to items of the database, and a matching score; For at least a subset of the digital teeth stored in the digital library, performing the steps c) to e), including c) retrieving a second limited set of parameters of the tooth selected from the digital database (from Cofar) and labelling image elements of the colour image within a region having a colour or colour pattern corresponding either to (i) a surface colour or surface colour pattern of a utensil used intraorally while obtaining the scanning dataset or (ii) to a colour pattern corresponding to a colour pattern of a tooth surface area comprising undesired stains or particles (from Dawood), in order to provide search through library data of a plurality of reference objects in a library to identify that a surface pattern of the reference object in the image data matches a surface pattern of the reference object in the library data for the reference object. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663
Prosecution Insights