DETAILED ACTION
This office action is responsive to applicant’s communication filed 01/02/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Application is acknowledged as claiming priority to foreign application with application number KR10-2022-0058974 dated 05/13/2022. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Response to Arguments
Applicant’s arguments, see pg. 6-7, filed 01/02/2026, with respect to the rejection(s) of claim(s) 1 and 10 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Hétroy et al. ("Simple flexible skinning based on manifold modeling"). Hétroy et al. teaches mesh deformation based on the movement of a skeleton (hard tissue), where the assignment of vertex weights is based on the geodesic distance, or the shortest path between vertices taking into account the edge lengths (see “Claim Interpretation” section).
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
The following terms in the claims have been given the following interpretations in light of the specification:
“Depths of connection between vertices”:
The specification states: “In an embodiment, different weights may be assigned to the plurality of vertices according to a depth of connection between adjacent vertices. Here, the weights may be understood as weights generated by a force. For example, referring to FIG. 11, a weight assigned to a part (e.g., a red vertex) at which a force is directly generated may be 100%, and small weights that decrease gradually (70%, 40%, and 0%) may be assigned according to depths of connection between vertices.
The example provided in the specification suggests that the depth of connection is a discrete number of edges between vertices – in other words, the shortest path between vertices in an unweighted or unit-weighted graph. However, the specification does not explicitly state that the edges are unweighted/unit-weighted, or that the length of the edges between vertices is not taken into account. Therefore, the broadest reasonable interpretation of “depths of connection between vertices” also includes measurements of the shortest path between vertices in a weighted graph, also known as the graph geodesic distance, or the shortest distance along the lengths of the edges.
Should applicant wish different definitions, Applicant should point to the portions of the specification that clearly show a different definition.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Choi et al. (US 20130328869 A1, hereinafter "Choi") in view of Jee et al. (WO 2016003255 A2, hereinafter "Jee"), Raslambekov (US 20210251723 A1), Hétroy et al. ("Simple flexible skinning based on manifold modeling". International Conference on Computer Graphics Theory and Applications (GRAPP), Feb 2009, Lisbon, Portugal. https://inria.hal.science/inria-00339413v1, hereinafter Hétroy), and Chen et al. (CN 102375908 A, hereinafter "Chen").
Regarding claim 1, Choi discloses: A method of performing a face transformation simulation (para. 0001 “…a method and system for providing a face adjustment image to display a predicted facial image to confirm an operation plan before performing a face correction operation by using face correction techniques such as surgical plastic surgery or orthodontic therapy”), which is performed by one or more processors of a computing device (para. 0026 “According to still another aspect of the present invention, there is provided a computer system for providing a face adjustment image…”), the method comprising:
obtaining volume data (para. 0026 “a first image acquisition unit to acquire a two-dimensional cephalometric image having a cranium image of a patient whose face is to be corrected”) and 3D facial data (para. 0026 “a second image acquisition unit to acquire a three-dimensional facial image by photographing the face of the patient”);
generating matching data by matching the volume data and the 3D facial data (para. 0012 “generating a matched image by superimposing a cephalometric image having a cranium image of a patient whose face is to be corrected with a three-dimensional facial image of the patient”);
generating a user interface including the matching data and providing the user interface to a user terminal (fig. 15, para. 0047 “FIG. 15 is a captured image of a display window showing the simulation procedures to generate a predicted facial image using a matched image.”); and
performing transformation on the matching data on the basis of a response to the user interface, the response being received from the user terminal (para. 0129 “When the user selects the Teeth checkbox on the right side of the screen, a teeth model, which is superimposed with the corresponding teeth part of the cephalometric image in the process of arranging the landmark points and lines of the cephalometric image, is shown in the form of the outline forming the edge of the teeth. When the user changes the location and orientation of the teeth model such as an upper/lower incisor model in the cephlometric image of the matched image using a mouse, the change in the soft skin tissues according to the change in the teeth model is displayed on the screen.”),
wherein the performing of the transformation comprises:
changing, based on a hard tissue adjustment input received by the user interface, a hard tissue (para. 0129 “When the user changes the location and orientation of the teeth model such as an upper/lower incisor model in the cephlometric image of the matched image using a mouse…”); and
changing a soft tissue corresponding to the hard tissue when the hard tissue is changed (para. 0129 “…the change in the soft skin tissues according to the change in the teeth model is displayed on the screen.”).
Choi does not explicitly teach that the volume data is three-dimensional (3D) volume data.
Jee teaches a similar method of performing a face transformation simulation comprising obtaining three-dimensional (3D) volume data (pg. 3 lines 101-102 “a third step in which a 3D oral surface model is created using tomography or a 3D scanning method”) and 3D facial data (pg. 3 lines 97-99 “a first step in which a 3D head image is created using a plurality of 2D tomographic image data obtained through tomography”).
Choi and Jee are both considered to be analogous to the claimed invention because they are in the same field of generating a 3D simulated preview of the results of a cosmetic medical procedure. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Choi to incorporate the teachings of Jee to use 3D volume data instead of 2D data to match the 3D facial data. The motivation would have been to simplify the matching procedure, as both sets of data would already be represented in 3D.
For the sake of conciseness, the same rationale will also apply to all mentions of “the 3D volume data” in further claims.
The combination of Choi in view of Jee does not explicitly teach: changing a soft tissue corresponding to the hard tissue using a weighted array when the hard tissue is changed, wherein the weighted array calculates a movement coefficient of the soft tissue according to the change in the hard tissue,
wherein the weighted array comprises a mesh including a plurality of vertices and a plurality of edges connecting the plurality of vertices,
wherein the plurality of vertices are matched to setting values according to a positional relationship with one or more landmarks,
wherein, when a force is applied to the plurality of vertices according to movement of the hard tissue, the weighted array moves the plurality of vertices on the basis of the applied force and the setting values matched to the plurality of vertices, and
wherein the plurality of vertices are assigned different weights according to depths of connection between adjacent vertices, and wherein a weight assigned to a vertex at which the force is directly generated is 100%, and smaller weights are gradually assigned to other vertices as the depths of connection between the adjacent vertices are deeper.
Raslambekov teaches: changing soft tissue corresponding to the hard tissue using a weighted array when the hard tissue is changed,
wherein the weighted array calculates a movement coefficient of the soft tissue according to the change in the hard tissue (para. 0026 “receiving a three-dimensional (3D) digital model of an archform comprising a representation of the gingiva and a plurality of teeth, the 3D model comprising a 3D mesh describing a surface of the gingiva and the plurality of teeth, the 3D mesh comprising a plurality of vertices connected to one another other by edges; identifying contour vertices defining a contour between a gingiva mesh describing portions of the 3D mesh corresponding to the gingiva and portions of the 3D mesh corresponding to a given tooth of the plurality of teeth; assigning, to the given tooth, a node representing a position of the given tooth; assigning a vertex weight equal to 1.0 to the contour vertices and to each vertex of the portions of the 3D mesh corresponding to the given tooth; determining a vertex weight, corresponding to the given tooth, for each remaining vertex of the gingiva mesh not belonging to the contour vertices by solving a harmonic equation describing a smooth gradation of vertex weights of the gingiva mesh, a boundary condition of the harmonic equation being linked to the vertex weight equal to 1.0 of the contour vertices, the vertex weight being representative of influence of movement of the node of the given tooth on a corresponding vertex of the gingiva mesh; applying a tooth-movement displacement to the node of the given tooth; determining a vertex-specific displacement for each vertex of the 3D mesh based on the vertex weight specific to each vertex of the 3D mesh and the tooth-movement displacement of the node; and updating the 3D model by applying the vertex-specific displacement to each corresponding vertex of the 3D mesh.”),
wherein the weighted array comprises a mesh including a plurality of vertices and a plurality of edges connecting the plurality of vertices (para. 0026 “receiving a three-dimensional (3D) digital model of an archform comprising a representation of the gingiva and a plurality of teeth, the 3D model comprising a 3D mesh describing a surface of the gingiva and the plurality of teeth, the 3D mesh comprising a plurality of vertices connected to one another other by edges”),
wherein the plurality of vertices are matched to setting values according to a positional relationship with one or more landmarks (para. 0026 “identifying contour vertices defining a contour between a gingiva mesh describing portions of the 3D mesh corresponding to the gingiva and portions of the 3D mesh corresponding to a given tooth of the plurality of teeth; assigning, to the given tooth, a node representing a position of the given tooth; assigning a vertex weight equal to 1.0 to the contour vertices and to each vertex of the portions of the 3D mesh corresponding to the given tooth; determining a vertex weight, corresponding to the given tooth, for each remaining vertex of the gingiva mesh not belonging to the contour vertices by solving a harmonic equation describing a smooth gradation of vertex weights of the gingiva mesh, a boundary condition of the harmonic equation being linked to the vertex weight equal to 1.0 of the contour vertices, the vertex weight being representative of influence of movement of the node of the given tooth on a corresponding vertex of the gingiva mesh”),
wherein, when a force is applied to the plurality of vertices according to movement of the hard tissue, the weighted array moves the plurality of vertices on the basis of the applied force and the setting values matched to the plurality of vertices (para. 0026 “determining a vertex-specific displacement for each vertex of the 3D mesh based on the vertex weight specific to each vertex of the 3D mesh and the tooth-movement displacement of the node; and updating the 3D model by applying the vertex-specific displacement to each corresponding vertex of the 3D mesh”), and
wherein the plurality of vertices are assigned different weights, and wherein a weight assigned to a vertex at which the force is directly generated is 100% ([0070] “If the vertex selected at step 325 corresponds to a tooth and/or the boundary between the tooth and gingiva, the vertex may be assigned a node weight indicating that the vertex will be shifted in the same manner as the entire tooth. In other words, a vector corresponding to the shift of the tooth will be applied directly to this vertex. The node weight may be set to ‘1.0,’ which indicates that a vector indicating movement of the node should not be scaled (reduced) when applied to this vertex.”), and smaller weights are gradually assigned to other vertices ([0074] “The node weights assigned to vertices may gradually decrease as the distance of the vertices from the tooth increase.”).
Choi, Jee, and Raslambekov are all considered to be analogous to the claimed invention because they are in the same field of generating a 3D simulated preview of the results of a cosmetic medical procedure. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the previously described invention of Choi in view of Jee to incorporate the teachings of Raslambekov to use a weighted mesh of vertices and edges to determine the movement of the gums based on the movement of the teeth. The motivation would have been to provide a numerical method of calculating the movement of the soft tissue which could accurately model the elastic behavior of the gums.
Raslambekov teaches that vertex weights are determined based on the distance from that vertex to the nearest vertex located on the hard tissue ([0071]). Raslambekov does not provide specifics on the type of distance calculation, or whether the distance is direct (Euclidean) or along the edges of the graph. Therefore, the combination of Choi in view of Jee and Raslambekov does not explicitly teach: wherein the plurality of vertices are assigned different weights according to depths of connection between adjacent vertices, and wherein a weight assigned to a vertex at which the force is directly generated is 100%, and smaller weights are gradually assigned to other vertices as the depths of connection between the adjacent vertices are deeper.
Hétroy teaches mesh deformation based on the movement of a skeleton (hard tissue), wherein the plurality of vertices are assigned different weights according to depths of connection between adjacent vertices (section 5 “Overlaps as well as skinning weights were computed with an approximated geodesic distance, using Dijkstra’s algorithm on the mesh’s vertices.”), and smaller weights are gradually assigned to other vertices as the depths of connection between the adjacent vertices are deeper (section 4 “This cubic function lets us define weights which decrease smoothly towards as the vertex vis closer to the region boundary”).
Hétroy is analogous to the claimed invention because it pertains to the same problem of simulating the deformation of a non-rigid material based on the movement of a connected hard material. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Choi in view of Jee and Raslambekov to incorporate the teachings of Hétroy to determine the vertex weights based on the graph geodesic distance. The motivation would have been to avoid the issues associated with using the Euclidean distance, which can “generate artifacts” around joints and curved regions (Hétroy section 5.2).
The combination of Choi in view of Jee, Raslambekov, and Hétroy does not explicitly teach that the hard tissue is changed through a plurality of surgical operations, wherein the plurality of surgical operations comprise:
a first surgical operation of performing an x-axis movement of the hard tissue;
a second surgical operation of performing a z-axis rotation of the hard tissue with respect to an arbitrary reference point, after the first surgical operation;
a third surgical operation of performing a y-axis rotation of the hard tissue with respect to the arbitrary reference point after, after the second surgical operation;
a fourth surgical operation of performing a z-axis movement of the hard tissue, after the third surgical operation;
a fifth surgical operation of performing an x-axis rotation of the hard tissue with respect to the arbitrary reference point after the fourth surgical operation; and
a sixth surgical operation of performing a y-axis movement of the hard tissue, after the fifth surgical operation.
Chen teaches that the hard tissue is changed through a plurality of surgical operations (para. 0011-0012 “providing at least one operator for moving the tooth in a discrete manner),
wherein the plurality of surgical operations comprise:
(para. 0016 “In one embodiment, the operator includes translation along the coordinate system X axis, translation along the coordinate system Y axis, translation along the coordinate system Z axis, rotation around the coordinate system X axis, rotation around the coordinate system Y axis, rotation around the coordinate system Z axis, and any combination thereof”)
a first surgical operation of performing an x-axis movement (para. 0016 “…translation along the coordinate system X axis…”);
a second surgical operation of performing a z-axis rotation with respect to an arbitrary reference point (para. 0016 “…rotation around the coordinate system Z axis…”) after applying the first surgery operation (para. 0016 “…and any combination thereof”) with respect to one or more landmarks (para. 0055 “In some embodiments, the digital data set includes coordinate data corresponding to a single tooth, and the digital data set includes coordinate data for all teeth in the tooth group”);
a third surgical operation of performing a y-axis rotation with respect to the arbitrary reference point (para. 0016 “…rotation around the coordinate system Y axis…”) after applying the second surgery operation (para. 0016 “…and any combination thereof”) with respect to the one or more landmarks (para. 0055 “In some embodiments, the digital data set includes coordinate data corresponding to a single tooth, and the digital data set includes coordinate data for all teeth in the tooth group”);
a fourth surgical operation of performing the z-axis movement (para. 0016 “…translation along the coordinate system Z axis…”);
a fifth surgical operation of performing the x-axis rotation with respect to the arbitrary reference point (para. 0016 “…rotation around the coordinate system X axis…”) after applying the fourth surgery operation (para. 0016 “…and any combination thereof…”) with respect to the one or more landmarks (para. 0055 “In some embodiments, the digital data set includes coordinate data corresponding to a single tooth, and the digital data set includes coordinate data for all teeth in the tooth group”); and
a sixth surgical operation of performing a y-axis movement (para. 0016 “…translation along the coordinate system Y axis…”).
Chen does not explicitly teach a procedure which is surgical in nature, but Chen, along with the claimed invention and the combination of the previously cited references, teaches a medical procedure which involves adjusting the position and orientation of hard tissue. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Choi in view of Jee, Raslambekov, and Hétroy to incorporate the teachings of Chen to support a cosmetic surgical procedure which contains this specific sequence of operations. The motivation would have been to perform a comprehensive adjustment of hard tissue across all axes and orientations.
Regarding claim 2, the combination of Choi in view of Jee, Raslambekov, Hétroy, and Chen teaches the method of claim 1, wherein the 3D volume data is volume data related to a head and neck (Choi fig. 6) of a person who undergoes surgery (Choi para. 0012 “a cephalometric image having a cranium image of a patient whose face is to be corrected”) and comprises one or more landmarks of the head and neck of the person (Choi fig. 7 shows labeled landmark points, para. 0093 “the cephalometric image is loaded on the screen by the image display module and then the landmark points are displayed on the cephalometric image on the screen”), and
the 3D facial data is image data related to the face of the person who undergoes the surgery (Choi para. 0012 “a three-dimensional facial image of the patient”) and comprises polygon data and texture data (Choi para. 0087 “The 3D face model, i.e., the 3D facial image is generally comprised of meshes and texture images”).
Regarding claim 3, the combination of Choi in view of Jee, Raslambekov, Hétroy, and Chen teaches the method of claim 1, wherein the generating of the matching data comprises:
identifying landmarks for matching in the 3D volume data and the 3D facial data (Choi para. 0069 “More specifically, the first alignment points comprise a pair of facial alignment points such as the pronasale (the first pronasale) and the gnathion (the first gnathion) on the facial image, and the second alignment points comprise a pair of outline alignment points such as the pronasale (the second pronasale) and the gnathion (the second gnathion) on the cephalometric image, which are arranged on the positions corresponding to the facial alignment points, i.e., on the same positions of the face”); and
generating the matching data on the basis of the identified landmarks for matching (Choi para. 0013 “a plurality of first alignment points arranged on the facial image to superimpose the facial image and the cephalometric image are matched with a plurality of second alignment points arranged on the positions corresponding to those of the first alignment points on the outline of the cephalometric image, which is formed by the soft skin tissues, so that the facial image and the cephalometric image are superimposed”).
Regarding claim 4, the combination of Choi in view of Jee, Raslambekov, Hétroy, and Chen teaches the method of claim 1, wherein the user interface comprises:
a simulation selection screen for receiving a selection input related to an orthodontic simulation or a plastic and orthognathic surgery simulation from a user (Choi para. 0083 “Referring to FIG. 3, the user pushes the button to load the facial image (Load Face) in the user interface displayed on the screen by the interface display module 150, and then selects the 3D facial image data having the desired shape from the image data box so that the 3D face model, i.e., the facial image is displayed on the screen by the image display module 130”);
a matching data display screen displaying the matching data (Choi figs. 12 and 14, para. 0044 “FIG. 12 is a captured image of a display window showing a matched image in which the 3D facial image excluding the profile line is displayed transparently.”, para. 0046 “FIG. 14 is a captured image of a display window showing a half-cut matched image as one example of a matched image.”); and
a hard tissue change information input screen for receiving the hard tissue adjustment input in relation to the change in the hard tissue from the user (Choi para. 0129 “When the user selects the Teeth checkbox on the right side of the screen, a teeth model, which is superimposed with the corresponding teeth part of the cephalometric image in the process of arranging the landmark points and lines of the cephalometric image, is shown in the form of the outline forming the edge of the teeth. When the user changes the location and orientation of the teeth model such as an upper/lower incisor model in the cephlometric image of the matched image using a mouse, the change in the soft skin tissues according to the change in the teeth model is displayed on the screen.”).
Regarding claim 9, the combination of Choi in view of Jee, Raslambekov, Hétroy, and Chen teaches an apparatus for performing the method of claim 1, comprising:
a memory storing one or more instructions (Choi [0027] “According to yet another aspect of the invention, there is provided a computer-readable recording medium having stored thereon a program for providing a face adjustment image…); and
a processor configured to execute the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform the method of claim 1 (Choi [0027] “…which enables a computer to function as matching means to generate a matched image by superimposing a cephalometric image having a cranium image of a patient whose face is to be corrected with a three-dimensional facial image of the patient, and image display means to display a predicted facial image on a screen based on the transformation of soft skin tissues in the matched image.”).
Regarding claim 10, it is rejected using the same references, rationale, and motivations to combine as claim 1, with the additional limitation of a non-transitory computer-readable recording medium on which a program for executing a method of performing a face transformation simulation in conjunction with a computing device is recorded (Choi [0027] “According to yet another aspect of the invention, there is provided a computer-readable recording medium having stored thereon a program for providing a face adjustment image, which enables a computer to function as matching means to generate a matched image by superimposing a cephalometric image having a cranium image of a patient whose face is to be corrected with a three-dimensional facial image of the patient, and image display means to display a predicted facial image on a screen based on the transformation of soft skin tissues in the matched image.”).
References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hutchinson et al. (US 20140267252 A1) teaches deformation of a mesh representing soft tissue, where the assignment of vertex weights is based on the geodesic distance, or the shortest path between vertices taking into account the edge lengths.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/ Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/ Supervisory Patent Examiner, Art Unit 2611