DETAILED ACTION
This office action is responsive to original claims filed on 05/30/2023. Presently, Claims 1 - 17 remain pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 3, 13, and 15-16 are objected to because of the following informalities:
Claim 3, Line 10, recites “smallest eigenvalue λ”, which should be changed to “smallest eigenvalue λ3”.
Claim 13, Line 2, recites “the convolutional neural network”, which should be changed to “the convolutional neural network model”.
Claim 15, Line 2, recites “heat map”, which should be changed to “map”, “probability map” or any term that may be suitable. The application is not related to “heat”.
Claim 16, Line 2, recites “heat map”, which should be changed to “map”, “probability map” or any term that may be suitable. The application is not related to “heat”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1, Lines 4-6, recites “determining full arch data … and partial arch data … by applying the 2D depth image to a convolutional neural network model”. It is unclear whether as the limitation of “by applying the 2D depth image to a convolutional neural network model” refers to both the determination of full arch data and partial arch data, or the determination of one of full arch data and partial arch data. For present purposes of examination, Examiner interprets the recited “by applying the 2D depth image to a convolutional neural network model” to determine one of full arch data and partial arch data, but not both of them.
Claim 1, Lines 7-8, recites “fully-connected convolutional neural network”. In the field of artificial neural network, the terms “fully-connected neural network” and “convolution neural network” are widely used, a convolutional neural network may also contain fully-connected layers, but the above-recited term is not used. Specifically, there is no such network that can be fully-connected neural network and convolution neural network at the same time. Hence, it is unclear how the recited term “fully-connected convolutional neural network” differs from “convolutional neural network”. For present purposes of examination, Examiner interprets the recited term “fully-connected convolutional neural network” to refer to “convolutional neural network”.
Claim 9, Lines 3-4 and 6-7, Claim 10, Lines 1-3, Claim 11, Lines 2-3, 5, 7 and 9, Claim 12, Lines 2-5, and Claim 14, Lines 1-2, recite “fully-connected convolutional neural network”, which has the same issue as discussed above for Claim 1, Lines 7-8, and are interpreted similarly.
Claim 13, Lines 1-2, recites “the detecting the 2D landmark further comprises training the convolutional neural network”, and Line 3, “the convolutional neural network”. The independent Claim 1, Lines 7-8, recites “detecting a 2D landmark … a fully-connected convolutional neural network model”. For present purposes of examination, the recited “the convolutional neural network” in Claim 13 is interpreted to refer to “the fully-connected convolutional neural network”.
Claims 2-8 and 15-17 are also rejected under 35 U.S.C. 112(b) because they inherit the indefiniteness of the claim(s) they respectively depend upon.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With regard to Claims 1-16:
Step 1: the claim is drawn to a method/process, one of the four statutory categories.
Step 2A, Prong One:
The claims recite the limitations of “projecting 3D scan data”, “determining full arch data … and partial arch data …”, “detecting a 2D landmark”, and “back-projecting the 2D landmark” in Claim 1, “determining a projection direction vector by a principal component analysis” in Claim 2, “moving a matrix”, “calculating a covariance”, “operating eigen decomposition”, and “determining the projection direction vector” in Claim 3, “determining w3 as the projection direction vector when …”, and “determining -w3 as the projection direction vector when …” in Claim 4, “the 2D depth image is generated on a projection plane”, and “the projection plane is defined at a location …” in Claim 5, and “the 2D landmark is back-projected in a direction …” in Claim 6, which are, under their broadest reasonable interpretation, limitations that cover performance of the limitation in the mind or mathematical calculations. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or mathematical calculations, then it falls within the “Mental Processes” grouping or “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements – convolutional neural network model in Claims 1, 7 and 13, fully-connected neural network model in Claims 1, 9-12 and 14, feature extractor comprising convolution layer and pooling layer in Claims 7-8, convolution process and deconvolution process in Claims 10-12 and 14, and training the convolutional neural network in Claim 13. The recited neural networks and their components or process are recited at a high-level of generality (i.e., as a generic artificial neural network performing a generic function of distinguishing types of images or detecting objects in images) such that it amounts no more than mere instructions to apply the exception using a generic neural network. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than mere instructions to apply a generic artificial neural network performing a generic function of distinguishing types of images or detecting objects in images. Mere instructions to apply an exception using a generic artificial neural network cannot provide an inventive concept.
For the reasons set forth above, Claims 1-16 are not patent eligible.
With regard to Claim 17:
Step 1: the claim is drawn to a device/system, one of the four statutory categories.
Step 2A, Prong One:
The claim is dependent on Claim 1, so recites the limitations of “projecting 3D scan data”, “determining full arch data … and partial arch data …”, “detecting a 2D landmark”, and “back-projecting the 2D landmark” in Claim 1, which are, under their broadest reasonable interpretation, limitations that cover performance of the limitation in the mind or mathematical calculations. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or mathematical calculations, then it falls within the “Mental Processes” grouping or “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claim is dependent on Claim 1, so recites the additional elements – convolutional neural network model in Claims 1, fully-connected neural network model in Claims 1, and non-transitory computer-readable storage medium and one hardware processor in Claim 17. The recited neural networks, storage medium and hardware processor are recited at a high-level of generality (i.e., as a generic artificial neural network stored in a generic storage medium, performing a generic function of distinguishing types of images or detecting objects in images as executed by a generic processor) such that it amounts no more than mere instructions to apply the exception using a generic neural network on a generic computer. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than mere instructions to apply a generic artificial neural network performing a generic function of distinguishing types of images or detecting objects in images on a generic computer. Mere instructions to apply an exception using a generic artificial neural network cannot provide an inventive concept.
For the reasons set forth above, Claim 17 is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 9, 13 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chung et al (IEEE Trans Med Imaging. 39(12):3900-3909; hereafter Chung), in view of Takahashi et al (J Prosthodont Res. 65(1):115-118; hereafter Takahashi).
With regard to Claim 1, Chung discloses a method for automatically detecting a landmark (the point and line pair) in three-dimensional (3D) dental scan data (Chung, Page 3902, Column 1, Para 1; “The point and line pairs are subsequently reconstructed (i.e., positioned) in the 3D domain. As the projected bounding plane, pertaining to the scanned model, is originally defined in a 3D domain, the point and line in Mp are automatically positioned in the 3D space.”. Chung discloses a method (termed as “deep pose regression”) for automatically detecting a landmark in 3D dental scan data. The method is also shown in Fig. 1(a).), the method comprising:
projecting 3D scan data (the scanned model) to generate a two-dimensional (2D) depth image (a synthetic depth image) (Chung, Page 3901, Column 2, Para 5; “For the scanned model, a synthetic depth image is generated for the primary axis (i.e., full-arch visible axis).”. The content in the left-upper corner of Fig. 1(a) (Page 3902) demonstrates the projecting procedure.);
detecting a 2D landmark in the 2D depth image using a fully-connected convolutional neural network model (Chung, Page 3902, Column 1, Para 1; “Then, the trained CNN model is used to acquire corresponding point and line pairs for each image (Fig. 1a).”); and
back-projecting the 2D landmark onto the 3D scan data to detect a 3D landmark of the 3D scan data (Chung, Page 3902, Column 1, Para 1; “The point and line pairs are subsequently reconstructed (i.e., positioned) in the 3D domain. … the point and line in Mp are automatically positioned in the 3D space.” Once the landmark detected in 2D image is positioned in 3D image, the landmark becomes 3D landmark.).
Chung does not clearly and explicitly disclose the method comprising determining full arch data obtained by scanning all teeth of a patient and partial arch data obtained by scanning only a part of teeth of the patient by applying the 2D depth image to a convolutional neural network model.
Takahashi in the same field of endeavor discloses the method comprising determining full arch data obtained by scanning all teeth of a patient and partial arch data obtained by scanning only a part of teeth of the patient (Takahashi, Page 116, Column 1, Para 1; “… 1184 oral photographic images … consisted of four types of dental arches: edentulous, arches with posterior tooth loss (distal extension missing), arches with bounded edentulous space (intermediate missing), and intact dentition (without missing) in each jaw.”. Of the 4 listed arch types, the first 3 are partial arch, and the 4th is full arch.) by applying the 2D depth image to a convolutional neural network model (Takahashi, Abstract; “The purpose of this study was to develop a method for classifying dental arches using a convolutional neural network (CNN) …”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung, as suggested by Takahashi, in order to determine an image to be full arch or partial arch. One of ordinary skill in the art would have been motivated to make the modification for the benefit of improved accuracy of the detected landmark by using different detection models for different arch types.
With regard to Claim 2, Chung and Takahashi disclose the method of Claim 1. Chung further discloses wherein the projecting the 3D scan data comprises determining a projection direction vector (the primary axis) by a principal component analysis (Chung, Page 3901, Column 2, Para 5; “For the scanned model, a synthetic depth image is generated for the primary axis (i.e., full-arch visible axis). The primary axis can be easily obtained by principal component analysis (PCA) …”).
With regard to Claim 3, Chung and Takahashi disclose the method of Claim 2. Chung further discloses wherein the determining the projection direction vector comprises:
moving (
X
'
=
X
-
X
-
) (Chung, Page 3901, Column 2, Para 5; “… covariance matrix …
C
=
1
V
∑
i
(
v
i
-
u
)
(
v
i
T
-
u
T
)
…”. Chung discloses the moving of the matrix V (corresponding to X of Application) in the calculation of the covariance matrix (i.e. vi - u)) a matrix
X
=
x
1
x
2
…
x
n
y
1
y
2
…
y
n
z
1
z
2
…
z
n
of a set
i
∈
1,2
,
…
,
n
p
i
(
x
i
,
y
i
,
z
i
)
of coordinates of n 3D points of the 3D scan data (Chung, Page 3901, Column 2, Para 5; “In a given triangular mesh model M = {V, E}, where V and E are sets of vertices and edges, let a 3D vector v ∈ V be a positional vector in set V.”) based on an average value
X
-
of
X
=
x
1
x
2
…
x
n
y
1
y
2
…
y
n
z
1
z
2
…
z
n
(Chung, Page 3901, Column 2, Para 5; “Defining the mean vector as
u
=
1
V
∑
i
v
i
…”);
calculating a covariance
Σ
=
c
o
v
(
X
'
)
=
1
n
-
1
X
'
X
'
T
for the coordinates of the n 3D points (Chung, Page 3901, Column 2, Para 5; “… covariance matrix …
C
=
1
V
∑
i
(
v
i
-
u
)
(
v
i
T
-
u
T
)
…”.);
operating (
ΣA
=
A
Λ
) eigen decomposition of Σ (Chung, Page 3901, Column 2, Para 5; “PCA can be subsequently performed through eigen decomposition or singular value decomposition …”); and
determining the projection direction vector based on a direction vector w3 having the smallest eigenvalue λ among w1 = { w1p, w1q, w1r}, w2 = { w2p, w2q, w2r}, w3 = { w3p, w3q, w3r}, where
A
=
w
1
p
w
2
p
w
3
p
w
1
q
w
2
q
w
3
q
w
1
r
w
2
r
w
3
r
and
Λ
=
λ
1
0
0
0
λ
2
0
0
0
λ
3
(Chung, Page 3901, Column 2, Para 5; “The depth image, Mp, is then generated by projecting all vertices to a tight bounding plane that has v2 as a normal vector.”. Here the disclosed vector v2 is the eigen vector corresponding to the smallest eigen value).
With regard to Claim 4, Chung and Takahashi disclose the method of Claim 3. Chung further discloses wherein the determining the projection direction vector comprises:
determining w3 as the projection direction vector when
η
-
is an average of normal vectors of the 3D scan data and
w
3
∙
η
-
>
0
; and
determining -w3 as the projection direction vector when
η
-
is an average of normal vectors of the 3D scan data and
w
3
∙
η
-
≤
0
(According to specification of Application (Para 0059; “When the teeth protrude upward, the average of the normal vectors of the set of the triangles of the 3D scan data represents an upward direction. In contrast, when the teeth protrude downward, the average of the normal vectors of the set of the triangles of the 3D scan data generated a downward direction.”), the average of normal vectors of the 3D scan data corresponds to the direction from tooth root to occlusal surface. So the limitations in current claim require the projection to be along the direction. Chung discloses a projection direction vector along “full-arch visible axis” in Page 3901, Column 2, Para 5; “For the scanned model, a synthetic depth image is generated for the primary axis (i.e., full-arch visible axis).”. In addition, in Fig. 1a (partially shown below), the projecting in Chung is along the direction from tooth root to occlusal surface (see the red arrows), agreeing with the claim limitations).
PNG
media_image1.png
309
570
media_image1.png
Greyscale
Chung, Part of Fig. 1a
With regard to Claim 5, Chung and Takahashi disclose the method of Claim 2. Chung further discloses wherein the 2D depth image (depth image Mp) is generated on a projection plane (a tight bounding plane), and the projection plane is defined at a location separated by a predetermined distance from the 3D scan data (In left-upper corner of Fig. 1a (also shown in discussion with Claim 4), the projected depth image and the 3D model are separated by a distance) with the projection direction vector (v2) as a normal vector (Chung, Page 3901, Column 2, Para 5; “The depth image, Mp, is then generated by projecting all vertices to a tight bounding plane that has v2 as a normal vector.”).
With regard to Claim 6, Chung and Takahashi disclose the method of Claim 2. Chung further discloses wherein the 2D landmark is back-projected in a direction opposite to the projection direction vector onto the 3D scan data to detect the 3D landmark (Chung, Page 3902, Column 1, Para 1; “The point and line pairs are subsequently reconstructed (i.e., positioned) in the 3D domain. … the point and line in Mp are automatically positioned in the 3D space.”).
With regard to Claim 9, Chung and Takahashi disclose the method of Claim 1. Chung further discloses wherein the detecting the 2D landmark comprises:
detecting the 2D landmark using a first fully-connected convolutional neural network model (Chung, Page 3902, Column 2, Para 1; “For training and inference, we used the traditional VGG-16 network [47] (Fig. 2) …”) trained using full arch training data when the 2D depth image is the full arch data (Chung, Page 3902, Column 1, Para 1; “Then, the trained CNN model is used to acquire corresponding point and line pairs for each image (Fig. 1a).” Fig. 4 (b) and (c) shows that for the depth image Mp of the scanned model, full arch images are used for landmark detection.).
Chung and Takahashi as discussed above do not disclose detecting the 2D landmark using a second fully-connected convolutional neural network model trained using partial arch training data when the 2D depth image is the partial arch data.
Chung further discloses detecting the 2D landmark using a second fully-connected convolutional neural network model (Chung, Page 3902, Column 2, Para 1; “Two identical neural networks are used to train each projection image (i.e., Mp and Ip). The only difference is the final output tensor.” A second network is trained for detecting landmark in the maximum intensity projection image (Ip) of CT data.) trained using partial arch training data when the 2D depth image is the partial arch data (Chung, “A maximum intensity projection (MIP) image with respect to the x-axis direction, Ip, is generated from I (indicated by yellow arrows in Fig. 1a).”. As the projection of the 3D CT data is along x-axis, the resulted 2D depth image (Ip) is partial arch data.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung and Takahashi, as further suggested by Chung, in order to use a second network for detecting landmark in partial-arch image. One of ordinary skill in the art would have been motivated to make the modification for the benefit of optimized performance for landmark detection by using different networks for analyzing images that contain different amounts of features.
With regard to Claim 13, Chung and Takahashi disclose the method of Claim 1. Chung further discloses wherein the detecting the 2D landmark further comprises training the convolutional neural network (Chung, Page 3902, Column 2, Para 1; “For training and inference, we used the traditional VGG-16 network …”),
wherein the training the convolutional neural network comprises receiving a training 2D depth image and user-defined landmark information (Chung, Page 3902, Column 1, Para 2; “We manually annotated (i.e., determined the point and angle of the line) images for Mp and Ip. The overall loss is … where X, p, and θ are the input 2D image and ground-truth 2D point and angle of the line, respectively.” This disclosure shows that both the training 2D depth image and the user-defined landmark information are received and input into a loss function), and
wherein the user-defined landmark information includes a type of a training landmark and correct location coordinates of the training landmark in the training 2D depth image (Chung, Page 3902, Column 1, Para 2; “We manually annotated (i.e., determined the point and angle of the line) images for Mp and Ip. The overall loss is … where X, p, and θ are the input 2D image and ground-truth 2D point and angle of the line, respectively.”).
With regard to Claim 17, Chung and Takahashi disclose the method of Claim 1. Chung further discloses a non-transitory computer-readable storage medium having stored thereon at least one program comprising commands, which when executed by at least one hardware processor (Chung, Page 3902, Column 2, Para 1; “We trained the network for 100 epochs using an Intel i7-7700K desktop system with 4.2 GHz processor, 32 GB of memory and NVIDIA TITAN Xp GPU machine.”. Page 3906, Column 1 shows that the other analysis steps of the method are also performed on the same computer system).
Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Chung, in view of Takahashi as evidenced by He et al (arXiv 2015: 1512.03385; hereafter He).
With regard to Claim 7, Chung and Takahashi disclose the method of Claim 1, but do not clearly and explicitly disclose wherein the convolutional neural network model comprises:
a feature extractor configured to extract a feature of the 2D depth image; and
a classifier configured to calculate a score for arch classification information based on the feature extracted by the feature extractor.
Takahashi as evidenced by He further discloses wherein the convolutional neural network model (Takahashi, Page 116, Column 1, Para 2; “The ImageNet pretrained model of a 152-layer residual network model (ResNet152) [9] with fine tuning was used for preprocessing, and the dataset was trained to classify the type of dental arch.”) comprises:
a feature extractor configured to extract a feature of the 2D depth image (He, Page 3, Column 2, Para 6; “the convolutional layers … The network ends with a global average pooling layer and …”. The convolutional layers and the pooling layer combine to constitute a feature extractor. The layers are also shown in Fig. 3, right column. Note that the example shown in Fig. 3 is a 34-layer network. ResNet152, which contains 152 layers, has the same architecture.); and
a classifier configured to calculate a score for arch classification information based on the feature extracted by the feature extractor (He, Page 3, Column 2, Para 6; “The network ends with a global average pooling layer and a 1000-way fully-connected layer with softmax.”. Here the “fully-connected layer with softmax” is a classifier.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung and Takahashi, as further suggested by Takahashi, in order to use feature extractor to extract features and use classifier for classification. One of ordinary skill in the art would have been motivated to make the modification for the benefit of such architecture of feature extractor and classifier being the most widely used architecture of convolutional neural network.
With regard to Claim 8, Chung and Takahashi disclose the method of Claim 7, but do not clearly and explicitly disclose wherein the feature extractor comprises:
a convolution layer including a process of extracting features of the 2D depth image; and
a pooling layer including a process of culling the extracted features into categories.
Takahashi as evidenced by He further discloses wherein the feature extractor comprises:
a convolution layer including a process of extracting features of the 2D depth image; and
a pooling layer including a process of culling the extracted features into categories (He, Page 3, Column 2, Para 6; “the convolutional layers … The network ends with a global average pooling layer and …”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung and Takahashi, as further suggested by Takahashi, in order to use convolutional layer and pooling layer in the feature extractor. One of ordinary skill in the art would have been motivated to make the modification for the benefit of such structure of concatenating convolutional layers and ending with a pooling layer being used in all types of convolutional neural network.
Claims 10-12 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chung and Takahashi, further in view of Zhou et al (arXiv 2018:1807.10165; hereafter Zhou).
With regard to Claim 10, Chung and Takahashi disclose the method of Claim 9. Chung further discloses wherein each of the first fully-connected convolutional neural network model and the second fully-connected convolutional neural network model operates:
a convolution process extracting a landmark feature from the 2D depth image (Chung, Page 3902, Column 2, Para 1; “For training and inference, we used the traditional VGG-16 network [47] (Fig. 2) with a minor modification in the final layer to output a 3D tensor (i.e., a 2D point and angle).”. Fig. 2 shows the model architecture where multiple convolution layers are included for extracting feature from input image.).
Chung and Takahashi do not clearly and explicitly disclose but do not clearly and explicitly disclose a deconvolution process adding landmark location information to the landmark feature.
Zhou in the same field of endeavor discloses a deconvolution process adding landmark location information to the landmark feature (Zhou, Page 3, Para 3; “UNet++ starts with an encoder sub-network or backbone followed by a decoder sub-network.”. In Fig. 1(a), the black arrows labelled with “Up-sampling” form a deconvolution process (or termed as decoder), which is the opposite process to the convolution process labelled with “Down-sampling” (or encoder). In the deconvolution process, extracted features are associated with the corresponding classes (e.g. landmark or not) determined in the training process). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung and Takahashi, as suggested by Zhou, in order to include a devolution process in the network model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of assigning classification result to each pixel of an image so as to accomplish image-based tasks such as object detection and segmentation in a more effective way.
With regard to Claim 11, Chung, Takahashi and Zhou disclose the method of Claim 10, but do not disclose wherein the convolution process and the deconvolution process are repeatedly operated in the first fully-connected convolution neural network model,
wherein the convolution process and the deconvolution process are repeatedly operated in the second fully-connected convolution neural network model, and
wherein a number of the repeated operation of the convolution process and the deconvolution process in the first fully-connected convolution neural network model is different from a number of the repeated operation of the convolution process and the deconvolution process in the second fully-connected convolution neural network model.
Zhou further discloses wherein the convolution process and the deconvolution process are repeatedly operated in the first fully-connected convolution neural network model (Zhou, Page 3, Fig. 1(c) shows that a first model “UNet++ L4” repeats the convolution process (the down-pointed arrows) and the deconvolution process (the up-pointed arrows) 4 times.),
wherein the convolution process and the deconvolution process are repeatedly operated in the second fully-connected convolution neural network model (Zhou, Page 3, Fig. 1(c) shows that a second model “UNet++ L3” repeats the convolution process (the down-pointed arrows) and the deconvolution process (the up-pointed arrows) 3 times.), and
wherein a number of the repeated operation of the convolution process and the deconvolution process in the first fully-connected convolution neural network model is different from a number of the repeated operation of the convolution process and the deconvolution process in the second fully-connected convolution neural network model (Zhou, Page 3, Fig. 1(c) shows that two models UNet++ L4 and UNet++ L3 have different numbers (4 and 3, respectively) of repeated operation of the convolution process and the deconvolution process.).
PNG
media_image2.png
321
324
media_image2.png
Greyscale
Zhou, Part of Fig. 1c
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung, Takahashi and Zhou, as further suggested by Zhou, in order to use network models of different number of repeated operations of convolution/deconvolution for analyzing full-arch and partial-arch data. One of ordinary skill in the art would have been motivated to make the modification for the benefit of better balancing between computation speed and detection accuracy by using network models of different complexity for analyzing images with different amounts of features (Zhou, Page 7, Para 2; “UNet++ L3 achieves on average 32.2% reduction in inference time while degrading IoU by only 0.6 points. More aggressive pruning further reduces the inference time but at the cost of significant accuracy degradation.”).
With regard to Claim 12, Chung, Takahashi and Zhou disclose the method of Claim 11, but do not disclose wherein the number of the repeated operation of the convolution process and the deconvolution process in the first fully-connected convolution neural network model is greater than the number of the repeated operation of the convolution process and the deconvolution process in the second fully-connected convolution neural network model.
Zhou further discloses wherein the number of the repeated operation of the convolution process and the deconvolution process in the first fully-connected convolution neural network model is greater than the number of the repeated operation of the convolution process and the deconvolution process in the second fully-connected convolution neural network model (Zhou, Page 3, Fig. 1(c) shows that the number of the repeated operation in a first model “UNet++ L4” is 4, which is greater than 3 for a second model “UNet++ L3”.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung, Takahashi and Zhou, as further suggested by Zhou, in order to analyze the full-arch data using a network model with more repeated operations of convolution/deconvolution process than the model for analyzing partial-arch data. One of ordinary skill in the art would have been motivated to make the modification for the benefit of better balancing between computation speed and detection accuracy by properly choosing network models of different complexity for analyzing images with different amounts of features (Zhou, Page 7, Para 2; “UNet++ L3 achieves on average 32.2% reduction in inference time while degrading IoU by only 0.6 points. More aggressive pruning further reduces the inference time but at the cost of significant accuracy degradation.”).
With regard to Claim 14, Chung and Takahashi disclose the method of Claim 1. Chung further discloses wherein the fully-connected convolutional neural network model operates:
a convolution process extracting a landmark feature from the 2D depth image (Chung, Page 3902, Column 2, Para 1; “For training and inference, we used the traditional VGG-16 network [47] (Fig. 2) with a minor modification in the final layer to output a 3D tensor (i.e., a 2D point and angle).”. Fig. 2 shows the model architecture where multiple convolution layers are included for extracting feature from input image.).
Chung and Takahashi do not clearly and explicitly disclose but do not clearly and explicitly disclose a deconvolution process adding landmark location information to the landmark feature.
Zhou in the same field of endeavor discloses a deconvolution process adding landmark location information to the landmark feature (Zhou, Page 3, Para 3; “UNet++ starts with an encoder sub-network or backbone followed by a decoder sub-network.”. In Fig. 1(a), the black arrows labelled with “Up-sampling” form a deconvolution process (or termed as decoder), which is the opposite process to the convolution process labelled with “Down-sampling” (or encoder). In the deconvolution process or decoder, extracted features are associated with the corresponding classes (e.g. landmark or not) determined in the training process). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung and Takahashi, as suggested by Zhou, in order to include a devolution process in the network model. One of ordinary skill in the art would have been motivated to make the modification for the benefit of assigning classification result to each pixel of an image so as to accomplish image-based tasks such as object detection and segmentation in a more effective way.
With regard to Claim 15, Chung, Takahashi and Zhou disclose the method of Claim 14, but do not clearly and explicitly disclose wherein a result of the deconvolution process is a heat map corresponding to the number of the 2D landmarks.
Zhou further discloses wherein a result of the deconvolution process is a heat map corresponding to the number of the 2D landmarks (Zhou, Page 6, Para 1; “… a 1x1 convolutional layer followed by a sigmoid activation function was appended to each of the target nodes: {x0,j І j ϵ {1,2,3,4}}. As a result, UNet++ generates four segmentation maps given an input image, which will be further averaged to generate the final segmentation map.” In the map, the detected landmarks would be displayed as region with intensity closer to 1, as a result of the operations of sigmoid activation function and averaging). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung, Takahashi and Zhou, as further suggested by Zhou, in order to output a heat map showing the detected landmarks. One of ordinary skill in the art would have been motivated to make the modification for the benefit of clearly displaying the location of detected landmarks, which may enable timely check of detection accuracy by a user.
With regard to Claim 16, Chung, Takahashi and Zhou disclose the method of Claim 15, but do not clearly and explicitly disclose wherein pixel coordinate having a largest value in the heat map represents a location of the 2D landmark.
Zhou further discloses wherein pixel coordinate having a largest value in the heat map represents a location of the 2D landmark (Zhou, Page 6, Para 1; “… a 1x1 convolutional layer followed by a sigmoid activation function was appended to each of the target nodes: {x0,j І j ϵ {1,2,3,4}}.” By using sigmoid activation function at the end of the model, the pixels that belong to the object to be detected is assigned to a high probability value, so in the resulted map, the detected landmark is shown as high intensity relative to the background. Fig. 2 shows multiple examples of object detection, where the detected objects are displayed as white). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung, Takahashi and Zhou, as further suggested by Zhou, in order to display the detected landmarks as high-intensity region in the heat map. One of ordinary skill in the art would have been motivated to make the modification for the benefit of clearly displaying the location of detected landmarks, which may enable timely check of detection accuracy by a user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHANG whose telephone number is (571)272-7172. The examiner can normally be reached Monday-Friday 8am-5pm E.T..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.Z./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798
Prosecution Insights