DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 6 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if the claims are incorporated into their corresponding independent claims. The following is a statement of reasons for the indication of allowable subject matter:
In regards to dependent claim 6, none of the cited prior art alone or in combination provides motivation to teach “wherein the projecting of the composite image onto the face of the patient includes: obtaining a real-time patient facial image from the second external device; searching for the facial feature point from the real-time patient facial image; and continuously calculating registration between the composite image and the facial feature point depending on a change in the real-time patient facial image, and wherein the calculating of the registration between the composite image and the facial feature point depending on the change in the real-time patient facial image continuously includes; performing primary registration on the composite image and the real-time patient facial image based on a portion corresponding to locations of eyes; and after performing the primary registration, performing secondary registration on the composite image and the real-time patient facial image based on a portion corresponding to a face other than the locations of the eyes” as the references only teach concepts for compositing images onto a patient’s face through feature matching and use of registration algorithms, however the references fail to explicitly disclose the specific process above for the iterative calculation of the registration between the composite image in relation to changes of the real-time patient facial image including secondary registration processes that take into account portions of the face other than eye location, in conjunction with the features of claim 1 with which it depends for the purpose of medical image registration for augmented reality.
In addition, there is no teaching, suggestion, or motivation found in the current references and none that can be inferred from the examiner’s own knowledge with respect to the current limitation.
In regards to dependent claim 14, this claim recites limitations similar in scope to claim 6, and thus is objected to based on the same rationale as provided above.
As allowable subject matter has been indicated, applicant's reply must either comply with all formal requirements or specifically traverse each requirement not complied with. See 37 CFR 1.111(b) and MPEP § 707.07(a).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3 and 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Seok (KR 2022-0126665 A, hereinafter referenced “Seok”) in view of Hoon (KR 2022-0019353 A, hereinafter referenced “Hoon”).
In regards to claim 1. Seok discloses an operating method of a medical image processing device using augmented reality (AR) (Seok, para [0007]), the method comprising:
-obtaining, from a first external device, a first image including blood vessel information and bone tissue information according to tomography (Seok, para [0042] and [0054]; Reference at para [0042] discloses additionally, the simulation device (100) can acquire a two-dimensional facial image of the subject (i.e. obtained first image). For example, the simulation device (100) can obtain an image captured from the shooting device (30) (i.e. first external device) that includes the entire face of the subject. Para [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data (i.e. obtained blood vessel information and bone tissue information according to tomography regarding the facial model of anatomical features));
-extracting a facial feature point from facial unique information including the blood vessel information, the bone tissue information, the facial structure, and the facial curve (Seok, para [0042], [0054], and [0056]; Reference at para [0042] discloses additionally, the simulation device (100) can acquire a two-dimensional facial image of the subject. For example, the simulation device (100) can obtain an image captured from the shooting device (30) that includes the entire face of the subject. Para [0054] discloses the simulation device (100) can acquire three-dimensional volume data in which CT data of the subject's facial area and STL data of the subject's tooth area are merged. Para [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data. Para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image (i.e. extracted facial feature points from facial unique information including the blood vessel information, the bone tissue information, the facial structure, and the facial curve regarding the facial model of anatomical features));
-registering the first image Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data (i.e. registering the first image based on the facial feature point), respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned);
-generating a composite image by performing 3D modeling on the first image Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned (i.e. generated composite image through 3D modeling of registered image));
-projecting the composite image onto a face of a patient (Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned (i.e. generated composite image projected onto face of patient regarding the subject));
-and displaying, on a projected image, a first layer corresponding to the blood vessel information and a second layer including the bone tissue information (Seok, para [0056]-[0057]; Reference at para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image. Para [0057] discloses according to one embodiment of the present invention, the mutual support (connection) relationship between bone tissue and soft tissue defined by the simulation device (100) may mean that the range of soft tissue of a subject affected by a predetermined change in the subject's bone tissue (e.g., pressure application, movement, tilting, etc.) and information regarding the effect of the predetermined change in bone tissue on the soft tissue within the determined range are defined for each mutually distinct bone tissue object of the facial area. For example, the simulation device (100) can predict changes in soft tissue corresponding to changes in bone tissue (e.g., rotation (in angle units), displacement (in length units), etc.) based on defined mutual support (connection) relationships (i.e. different layers interpreted as the bone and soft tissue which would contain blood vessel info)).
Seok does not explicitly disclose but Hoon teaches
-obtaining, from a second external device, a second image including a facial structure and a facial curve according to 3D imaging / registering (second image) (Hoon, para [0011] and [0013]; Reference at para [0011] discloses receiving a medical image captured of a part of a subject's body from a medical image server to reconstruct a three-dimensional image and extracting feature points from the reconstructed three-dimensional image; capturing a part of the subject's body using a depth camera mounted on the user terminal (i.e. second external device), acquiring three-dimensional data of the subject from the captured image, and extracting feature points from the three-dimensional data. Para [0013] discloses the above-mentioned part of the body includes a face, and the above-mentioned feature point may include at least one of the eyes, nose, and mouth of the subject being measured (i.e. a second image including a facial structure and a facial curve according to 3D imaging regarding subject’s facial features);
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
In regards to claim 2. Seok in view of Hoon teach the method of claim 1.
Seok does not explicitly disclose but Hoon teaches
-wherein the extracting of the facial feature point from the facial unique information includes: extracting at least one first feature point corresponding to a bone tissue and a blood vessel from the first image (Hoon, para [0031]; Reference discloses first, the first feature point extraction unit (210) receives a medical image taken of a part of the body of a subject to measurement from a medical image server (100), reconstructs a three-dimensional image from the received medical image, and extracts feature points from the reconstructed three-dimensional image);
-extracting at least one second feature point corresponding to a skin area for eyes, a nose, a mouth, and ears of a human body based on an anatomical feature from the second image (Hoon, para [0039]-[0040]; Reference at para [0039] discloses the second feature point extraction unit (230) obtains three-dimensional data of a subject to measurement from an image captured by a depth camera and extracts feature points from the three-dimensional data. Para [0040] discloses likewise, the feature point extracted by the second feature point extraction unit (230) includes at least one of the eyes, nose, and mouth of the subject being measured);
-and extracting the facial feature point based on the first feature point and the second feature point (Hoon, para [0041]; Reference discloses the image matching unit (240) creates an augmented reality image by matching the feature points extracted by the first feature point extraction unit (210) with the feature points extracted by the second feature point extraction unit (230)).
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
In regards to claim 3. Seok in view of Hoon teach the method of claim 2.
Seok does not explicitly disclose but Hoon teaches
-wherein the registering of the first image to the second image based on the facial feature point includes: registering the facial feature point based on a skin surface derived from the second image (Hoon, para [0054] and [0056]; Reference at [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned. Para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image).
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
In regards to claim 8. Seok in view of Hoon teach the method of claim 1.
Seok further discloses
-wherein the displaying of the first layer corresponding to the blood vessel information and the second layer including the bone tissue information on the projected image includes: displaying the first layer; and separately displaying the second layer at a point in time when the first layer is not displayed, or in a region where the first layer is not displayed (Seok, para [0055] and [0064]; Reference at [0055] discloses the simulation device (100) may have an algorithm for identifying anatomical feature points from each of the facial image and volume data. For example, an algorithm for identifying anatomical feature points may be an algorithm that identifies anatomical feature points by performing background difference-based object detection, artificial intelligence-based object detection, semantic region segmentation-based object detection, etc., based on pixel-by-pixel color information of a facial image. Para [0064] discloses for example, as shown in FIGS. 6a and 6b, the main area in which the shape, etc. changes by user input authorized by the user may be identified by colors, shading, highlights, etc. that are distinct from the rest of the area).
In regards to claim 9. Seok discloses a medical image processing device using AR (Seok, para [0007]) comprising:
-a memory configured to store a feature extraction module (Seok, para [0099]) configured to extract a feature part from an image (Seok, para [0054]; Reference discloses [Additionally, according to one embodiment of the present invention, the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data),
-an image registration module configured to register at least two images (Seok, para [0054]; Reference discloses the simulation device (100) (i.e. image registration module) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data (i.e. registering the image), respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned), an image processing module configured to perform image processing on data constituting an image according to a predefined operation, and an AR processing module configured to implement and display the processed image in augmented reality (Seok, para [0055] and [0074]; Reference at para [0055] discloses in this regard, the simulation device (100) may have an algorithm for identifying anatomical feature points from each of the facial image and volume data. For example, an algorithm for identifying anatomical feature points may be an algorithm that identifies anatomical feature points by performing background difference-based object detection, artificial intelligence-based object detection, semantic region segmentation-based object detection, etc., based on pixel-by-pixel color information of a facial image (i.e. image processing image processing on data constituting an image according to a predefined operation). Para [0074] discloses the model generation unit (130) (i.e. AR processing module) can construct a virtual 3D facial model of a subject by superimposing volume data and a facial image processed by the processing unit (120));
-and a processor including one or more cores and configured to control operations of the feature extraction module, the image registration module, the image processing module, and the AR processing module (Seok, para [0071]; Reference discloses referring to FIG. 7, the simulation device (100) may include a collection unit (110), a processing unit (120), a model generation unit (130), an input receiving unit (140), an analysis unit (150), and a display unit (160)), wherein the memory is configured to:
-obtain a first image including blood vessel information and bone tissue information (Seok, para [0042] and [0054]; Reference at para [0042] discloses additionally, the simulation device (100) can acquire a two-dimensional facial image of the subject (i.e. obtained first image). For example, the simulation device (100) can obtain an image captured from the shooting device (30) (i.e. first external device) that includes the entire face of the subject. Para [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data (i.e. obtained blood vessel information and bone tissue information regarding the facial model of anatomical features)),
-and wherein the processor is configured to: extract a facial feature point from facial unique information including the blood vessel information, the bone tissue information, the facial structure, and the facial curve Seok, para [0042], [0054], and [0056]; Reference at para [0042] discloses additionally, the simulation device (100) can acquire a two-dimensional facial image of the subject. For example, the simulation device (100) can obtain an image captured from the shooting device (30) that includes the entire face of the subject. Para [0054] discloses the simulation device (100) can acquire three-dimensional volume data in which CT data of the subject's facial area and STL data of the subject's tooth area are merged. Para [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data. Para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image (i.e. extracted facial feature points from facial unique information including the blood vessel information, the bone tissue information, the facial structure, and the facial curve regarding the facial model of anatomical features));
-register the first image Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data (i.e. registering the first image based on the facial feature point), respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned);
-generate a composite image by performing 3D-modeling on the first image Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned (i.e. generated composite image through 3D modeling of registered image));
-project the composite image onto a face of a patient (Seok, para [0054]; Reference discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned (i.e. generated composite image projected onto face of patient regarding the subject));
-and display, on a projected image, a first layer corresponding to the blood vessel information and a second layer including the bone tissue information (Seok, para [0056]-[0057]; Reference at para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image. Para [0057] discloses according to one embodiment of the present invention, the mutual support (connection) relationship between bone tissue and soft tissue defined by the simulation device (100) may mean that the range of soft tissue of a subject affected by a predetermined change in the subject's bone tissue (e.g., pressure application, movement, tilting, etc.) and information regarding the effect of the predetermined change in bone tissue on the soft tissue within the determined range are defined for each mutually distinct bone tissue object of the facial area. For example, the simulation device (100) can predict changes in soft tissue corresponding to changes in bone tissue (e.g., rotation (in angle units), displacement (in length units), etc.) based on defined mutual support (connection) relationships (i.e. different layers interpreted as the bone and soft tissue which would contain blood vessel info)).
Seok does not explicitly disclose but Hoon teaches
-and a second image including a facial structure and a facial curve from an external device / registering (second image) (Hoon, para [0011] and [0013]; Reference at para [0011] discloses receiving a medical image captured of a part of a subject's body from a medical image server to reconstruct a three-dimensional image and extracting feature points from the reconstructed three-dimensional image; capturing a part of the subject's body using a depth camera mounted on the user terminal (i.e. second external device), acquiring three-dimensional data of the subject from the captured image, and extracting feature points from the three-dimensional data. Para [0013] discloses the above-mentioned part of the body includes a face, and the above-mentioned feature point may include at least one of the eyes, nose, and mouth of the subject being measured (i.e. a second image including a facial structure and a facial curve according to 3D imaging regarding subject’s facial features);
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
In regards to claim 10. Seok in view of Hoon teach the medical image processing device of claim 9.
Seok does not explicitly disclose but Hoon teaches
-wherein when extracting the facial feature point, the processor is configured to: extract at least one first feature point corresponding to a bone tissue and a blood vessel from the first image (Hoon, para [0031]; Reference discloses first, the first feature point extraction unit (210) receives a medical image taken of a part of the body of a subject to measurement from a medical image server (100), reconstructs a three-dimensional image from the received medical image, and extracts feature points from the reconstructed three-dimensional image);
-extract at least one second feature point corresponding to a skin area for eyes, a nose, a mouth, and ears of a human body based on an anatomical feature from the second image (Hoon, para [0039]-[0040]; Reference at para [0039] discloses the second feature point extraction unit (230) obtains three-dimensional data of a subject to measurement from an image captured by a depth camera and extracts feature points from the three-dimensional data. Para [0040] discloses likewise, the feature point extracted by the second feature point extraction unit (230) includes at least one of the eyes, nose, and mouth of the subject being measured);
-and extract the facial feature point based on the first feature point and the second feature point (Hoon, para [0041]; Reference discloses the image matching unit (240) creates an augmented reality image by matching the feature points extracted by the first feature point extraction unit (210) with the feature points extracted by the second feature point extraction unit (230)).
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
In regards to claim 11. Seok in view of Hoon teach the medical image processing device of claim 10.
Seok does not explicitly disclose but Hoon teaches
-wherein when registering the first image to the second image based on the facial feature point, the processor is configured to: register the facial feature point based on a skin surface derived from the second image (Hoon, para [0054] and [0056]; Reference at [0054] discloses the simulation device (100) can construct the shape of a virtual 3D facial model of a subject by deriving the positions of a plurality of anatomical feature points (e.g., eyes, eyebrows, forehead, nose tip, chin tip, cheekbones, etc.) from a facial image and volume data, respectively, and overlaying the facial image on the surface of the 3D volume data so that the positions of the feature points derived from the facial image and volume data are aligned. Para [0056] discloses additionally, the simulation device (100) can define the mutual support (connection) relationship between the bone tissue of the facial area, including the subject's teeth, skull, maxilla, mandible, cheekbone, forehead bone, etc., identified from volume data, and the soft tissue of the subject's skin surface identified from the facial image).
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
Claims 4, 5, 7, 12, 13, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Seok (KR 2022-0126665 A) in view of Hoon (KR 2022-0019353 A) as applied to claims 1 and 9 above, and further in view of Hyuk (KR 2021-0051141 A, hereinafter referenced “Hyuk”).
In regards to claim 4. Seok in view of Hoon teach the method of claim 2.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein the extracting of the at least one second feature point includes: extracting a stable feature point within a face with little deformation; and adjusting the second feature point based on an outline and coordinates of the stable feature point, and wherein the stable feature point is a location and a size of eyes with respect to an entire facial contour (Hyuk, para [0058] and [0061]; Reference at para [0058] discloses that is, the accuracy of the alignment can be increased by first performing a primary alignment based on information about a reference skin where the alignment part (140) is relatively less deformed. Para [0061] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points corresponding to reference skin and a plurality of second feature points corresponding to reference skin using a Point Pair Feature (PPF) algorithm as a first matching algorithm).
Seok and Hoon are combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok to include the augmented reality medical image display features of Hoon in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality to provide better realism and additional information for surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok.
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
In regards to claim 5. Seok in view of Hoon teach the method of claim 1.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein the facial feature point serves as a marker used in image registration (Hyuk, para [0054]; Reference discloses information about the patient's skin includes location information of a plurality of first feature points constituting the patient's skin, and the location information of a plurality of first feature points constituting the reference skin can function as a marker).
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
In regards to claim 7. Seok in view of Hoon teach the method of claim 1.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein the registering of the first image to the second image based on the facial feature point includes: calculating approximate locations of at least one first feature point corresponding to a bone tissue and a blood vessel, and at least one second feature point corresponding to a skin area for eyes, a nose, a mouth, and ears of a human body from the second image based on a first registration algorithm; and calculating precise locations of the at least one first feature point and the at least one second feature point based on a second registration algorithm (Hyuk, para [0061] and para [0065]; Reference at para [0061] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points corresponding to reference skin and a plurality of second feature points corresponding to reference skin using a Point Pair Feature (PPF) algorithm as a first matching algorithm. Para [0065] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points and a plurality of second feature points corresponding to the reference skin, and then calculate the precise positions of a plurality of first feature points corresponding to the reference skin and a plurality of second feature points corresponding to the reference skin based on a second matching algorithm).
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
In regards to claim 12. Seok in view of Hoon teach the medical image processing device of claim 10.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein when extracting the at least one second feature point, the processor is configured to: extract a stable feature point within a face with little deformation; and adjust the second feature point based on an outline and coordinates of the stable feature point, and wherein the stable feature point is a location and a size of eyes with respect to an entire facial contour (Hyuk, para [0058] and [0061]; Reference at para [0058] discloses that is, the accuracy of the alignment can be increased by first performing a primary alignment based on information about a reference skin where the alignment part (140) is relatively less deformed. Para [0061] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points corresponding to reference skin and a plurality of second feature points corresponding to reference skin using a Point Pair Feature (PPF) algorithm as a first matching algorithm).
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
In regards to claim 13. Seok in view of Hoon teach the medical image processing device of claim 9.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein the facial feature point serves as a marker used in image registration (Hyuk, para [0054]; Reference discloses information about the patient's skin includes location information of a plurality of first feature points constituting the patient's skin, and the location information of a plurality of first feature points constituting the reference skin can function as a marker).
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
In regards to claim 15. Seok in view of Hoon teach the medical image processing device of claim 9.
Seok and Hoon does not explicitly disclose but Hyuk teaches
-wherein when registering the first image to the second image, the processor is configured to: calculate approximate locations of at least one first feature point corresponding to a bone tissue and a blood vessel, and at least one second feature point corresponding to a skin area for eyes, a nose, a mouth, and ears of a human body from the second image based on a first registration algorithm; and calculate precise locations of the at least one first feature point and the at least one second feature point based on a second registration algorithm (Hyuk, para [0061] and para [0065]; Reference at para [0061] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points corresponding to reference skin and a plurality of second feature points corresponding to reference skin using a Point Pair Feature (PPF) algorithm as a first matching algorithm. Para [0065] discloses the matching unit (140) can calculate the approximate positions of a plurality of first feature points and a plurality of second feature points corresponding to the reference skin, and then calculate the precise positions of a plurality of first feature points corresponding to the reference skin and a plurality of second feature points corresponding to the reference skin based on a second matching algorithm).
Seok and Hyuk are also combinable because they are in the same field of endeavor regarding medical image generation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for facial image simulation and matching system of Seok, in view of the augmented reality medical image display features of Hoon, to include the augmented reality medical information features of Hyuk in order to provide the user with a system for constructing a 3D facial model capable of performing virtual surgery and predicting surgical results by superimposing a subject's facial image and 3D radiographic image data as taught by Seok while incorporating the augmented reality medical image display features of Hoon to allow for use of a user terminal for providing augmented reality medical images that provide a three-dimensional image of each subject to measure together with an image of the subject to measure using augmented reality. Further incorporating the augmented reality medical information features of Hyuk allows for augmented reality based display of medical information that can improve accuracy of a coordinate system alignment through non-marker-based coordinate improving surgical assistance, applicable to medical/surgical modelling and simulation systems such as those taught in Seok and Hoon.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: See the Notice of References Cited (PTO-892)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TERRELL M ROBINSON whose telephone number is (571)270-3526. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KENT CHANG can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TERRELL M ROBINSON/Primary Examiner, Art Unit 2614
Prosecution Insights