DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
The amendments to claims 4, 7, 15, and 16 are accepted and entered.
Claim 14 is cancelled.
New claims 17-21 are accepted.
Claims 1-13 and 15-21 are pending.
The amendments to the Specification are accepted and entered.
Priority
The present application claims foreign priority benefits from CN202210475990.9 filed on
04/29/2022. The certified copies of the priority documents were electronically retrieved on 07/18/2025.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/02/2024 and 07/18/2025 are
considered and attached.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because signals per se do not fall into one of the four statutory categories.
Claim 16 recites, inter alia, “A storage medium comprising computer-executable instructions…” After close inspection, the Examiner respectfully notes that the disclosure, as a whole, does not definitively describe what can and cannot be considered the “storage medium”. Applicant’s specification discusses the “storage medium” in paragraph 0188. Applicant describes the “computer-readable storage medium” as possibly being tangible. However, while the recited examples of the mediums which may act as the “computer-readable storage medium” are all non-transitory examples, applicant specifically states that the examples of the computer-readable storage medium may include, but not be limited to “[see the examples as listed in para. [0188] of applicant’s specification]”. Thus, the “computer-readable storage medium” could be other forms – such as a signal.
An Examiner is obliged to give claims their broadest reasonable interpretation consistent with the specification during examination. The broadest reasonable interpretation of a claim drawn to a computer program product (also called a computer readable medium, machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal, per se, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter.
Therefore, given the non-definitive disclosure and the broadest reasonable interpretation, the machine-readable storage medium of the claim may include transitory propagating signals. As a result, the claim pertains to non-statutory subject matter.
However, the Examiner respectfully submits a claim drawn to such a computer program product or computer readable storage medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation “non-transitory” to the claim. Such an amendment would typically not raise the issue of new matter, even when the specification is silent because the broadest reasonable interpretation relies on the ordinary and customary meaning that includes signals per se. For additional information, please see the Patents’ Official Gazette notice published February 23, 2010 (1351 OG 212).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 12, 15, 16, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li.
Regarding claim 1, Cheng teaches an image segmentation method (Cheng, para. [0096]), comprising:
obtaining an image to be segmented (Cheng teaches “acquiring a first segmentation image corresponding to the image to be segmented” in para. [0096]);
determining a preliminarily segmented image (Cheng teaches “acquiring a first segmentation image corresponding to the image to be segmented, wherein the first segmentation image is formed by performing super segmentation on the image to be segmented, and the first segmentation image comprises a plurality of areas” in para. [0096]) and a target normal vector image (Cheng teaches “calculating the feature values of the features to be extracted of every two adjacent regions in the current first segmentation image, and obtaining the feature vectors of every two adjacent regions in the current first segmentation image according to a preset algorithm, wherein the feature vectors are vectors formed by the feature values of the features to be extracted, and the feature values in the feature vectors are used for representing the difference between the two adjacent regions” in para. [0107]) that are corresponding to the image to be segmented; and
performing image fusion on the preliminarily segmented image and the target normal vector image to obtain a target segmented image (Cheng teaches “the regions formed after the super-segmentation [the first segmentation/preliminarily segmented image] are merged for a preset number of times, each merging is performed by using the image formed by the region merging at the previous time as an input, and the steps S102 to S105 are required to be performed for each region merging” in para. [0101], wherein S103 involves determining the feature vectors, so therefore, the merging involves merging the first segmentation determined in s101 with the weighted feature vectors such that a target segmented image is created wherein “the features with the greatest importance in the region merging of this time are determined before the region merging of each time, so that the region merging of each time can be performed only according to the features with the greatest importance without paying attention to other features, and the processing time of the region merging is greatly saved on the basis of ensuring the accuracy of the region merging” as shown in para. [0106]. See also para. [0112] and [0122]).
While Cheng teaches the target vector image, Cheng fails to specifically teach the target normal vector image (emphasis added).
However, Li teaches a target normal vector image (Li teaches a segmentation method wherein “input data of the point cloud feature extraction network 312 […] may include other features, such as RGB colors and normal vectors, of each point cloud” in para. [0086]).
Cheng and Li are both considered to be analogous to the claimed invention because they are in the same field of segmenting objects by utilizing feature vectors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng to incorporate the teachings of Li and include “a target normal vector image”. The motivation for doing so would have been to expand the “representation of a point cloud feature including spatial coordinate information”, as suggested by Li in para. [0156]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng with Li to obtain the invention specified in claim 1.
Regarding claim 12, Cheng and Li teach the method according to claim 1, wherein the performing image fusion on the preliminarily segmented image and the target normal vector image to obtain the target segmented image comprises:
for each pixel in the preliminarily segmented image, determining a predicted weight of the pixel based on a predicted pixel value of the pixel in the target normal vector image, and a preset segmentation threshold (Li teaches “for each keypoint of an object, determining a weighted average value of a target predicted value corresponding to a keypoint and a point cloud of a preset value closest to a central point of the object and a probability value corresponding to the point cloud of the preset value in the instance mask information, as keypoint information of the object” as shown in para. [0143]. Here, the keypoints within the object are interpreted as equivalent to the pixels in the preliminarily segmented image. Additionally, the weighted average of the pixel (predicted weight) based on the target predicted value (predicted pixel value of the pixel in the target normal vector image) and point cloud of the preset value closest to a central point of the object (segmentation threshold) is interpreted as equivalent to the claim language above);
weighting a pixel value of the pixel in the preliminarily segmented image based on the predicted weight to obtain a target pixel value of the pixel (Li teaches “determining keypoint information of an object through regression based on the first offset and the instance segmentation information” as shown in para. [0123], wherein the keypoint information is interpreted as equivalent to the claimed target pixel value and the first offset/instance segmentation information and is based off of the weight generated by the predicted target pixel values. See also para. [0146]-[0149]); and
determining the target segmented image based on the target pixel value of each pixel in the preliminarily segmented image (Li teaches “the instance segmentation information determined based on the semantic segmentation information and the instance mask information may be used to specify a category of an object and location information of the object. A neural network 700 of FIG. 7 may perform end-to-end segmentation and 3D keypoint regression” in para. [0124]).
Similar motivation as applied to claim 1 can be applied here to claim 12.
Regarding claim 15, Cheng teaches an electronic device (Cheng, “device”, see abstract), comprising:
a processor (Cheng teaches a processor in para. [0052]); and
a storage apparatus, configured to store a program, wherein the program, when executed by the processor, causes the processor to implement an image segmentation method (Cheng teaches “a memory and a processor; the memory is used for storing program instructions, and the processor is used for calling the program instructions in the memory and executing the following method…” in para. [0052]-[0053]), wherein the method comprising:
obtaining an image to be segmented (Cheng teaches “acquiring a first segmentation image corresponding to the image to be segmented” in para. [0096]);
determining a preliminarily segmented image (Cheng teaches “acquiring a first segmentation image corresponding to the image to be segmented, wherein the first segmentation image is formed by performing super segmentation on the image to be segmented, and the first segmentation image comprises a plurality of areas” in para. [0096]) and a target (Cheng teaches “calculating the feature values of the features to be extracted of every two adjacent regions in the current first segmentation image, and obtaining the feature vectors of every two adjacent regions in the current first segmentation image according to a preset algorithm, wherein the feature vectors are vectors formed by the feature values of the features to be extracted, and the feature values in the feature vectors are used for representing the difference between the two adjacent regions” in para. [0107]) that are corresponding to the image to be segmented; and
performing image fusion on the preliminarily segmented image and the target (Cheng teaches “the regions formed after the super-segmentation [the first segmentation/preliminarily segmented image] are merged for a preset number of times, each merging is performed by using the image formed by the region merging at the previous time as an input, and the steps S102 to S105 are required to be performed for each region merging” in para. [0101], wherein S103 involves determining the feature vectors, so therefore, the merging involves merging the first segmentation determined in s101 with the weighted feature vectors such that a target segmented image is created wherein “the features with the greatest importance in the region merging of this time are determined before the region merging of each time, so that the region merging of each time can be performed only according to the features with the greatest importance without paying attention to other features, and the processing time of the region merging is greatly saved on the basis of ensuring the accuracy of the region merging” as shown in para. [0106]. See also para. [0112] and [0122]).
While Cheng teaches the target vector image, Cheng fails to specifically teach the target normal vector image (emphasis added).
However, Li teaches a target normal vector image (Li teaches a segmentation method wherein “input data of the point cloud feature extraction network 312 […] may include other features, such as RGB colors and normal vectors, of each point cloud” in para. [0086]).
Cheng and Li are both considered to be analogous to the claimed invention because they are in the same field of segmenting objects by utilizing feature vectors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng to incorporate the teachings of Li and include “a target normal vector image”. The motivation for doing so would have been to expand the “representation of a point cloud feature including spatial coordinate information”, as suggested by Li in para. [0156]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng with Li to obtain the invention specified in claim 15.
Regarding claim 16, Cheng and Li teach a storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, cause implementing the image segmentation method according to claim 1 (Cheng teaches “a memory and a processor” wherein “the memory is used for storing program instructions, and the processor is used for calling the program instructions in the memory” in para. [0052]-[0053]).
Regarding claim 21, Cheng and Li teach the electronic device according to claim 15, wherein the performing image fusion on the preliminarily segmented image and the target normal vector image to obtain the target segmented image comprises:
for each pixel in the preliminarily segmented image, determining a predicted weight of the pixel based on a predicted pixel value of the pixel in the target normal vector image, and a preset segmentation threshold(Li teaches “for each keypoint of an object, determining a weighted average value of a target predicted value corresponding to a keypoint and a point cloud of a preset value closest to a central point of the object and a probability value corresponding to the point cloud of the preset value in the instance mask information, as keypoint information of the object” as shown in para. [0143]. Here, the keypoints within the object are interpreted as equivalent to the pixels in the preliminarily segmented image. Additionally, the weighted average of the pixel (predicted weight) based on the target predicted value (predicted pixel value of the pixel in the target normal vector image) and point cloud of the preset value closest to a central point of the object (segmentation threshold) is interpreted as equivalent to the claim language above);
weighting a pixel value of the pixel in the preliminarily segmented image based on the predicted weight to obtain a target pixel value of the pixel (Li teaches “determining keypoint information of an object through regression based on the first offset and the instance segmentation information” as shown in para. [0123], wherein the keypoint information is interpreted as equivalent to the claimed target pixel value and the first offset/instance segmentation information and is based off of the weight generated by the predicted target pixel values. See also para. [0146]-[0149]); and
determining the target segmented image based on the target pixel value of each pixel in the preliminarily segmented image (Li teaches “the instance segmentation information determined based on the semantic segmentation information and the instance mask information may be used to specify a category of an object and location information of the object. A neural network 700 of FIG. 7 may perform end-to-end segmentation and 3D keypoint regression” in para. [0124]).
Similar motivation as applied to claim 15 can be applied her to claim 21.
Claims 2-5, 7, 9, and 17-20 rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li and Yao et al. (CN 113538480 A, see attached English translation for citations), hereinafter Yao.
Regarding claim 2, Cheng and Li teach the method according to claim 1.
While Cheng teaches obtaining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented (see claim 1), Cheng and Li fail to teach wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises: inputting the image to be segmented to an image segmentation model that has been pre- trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image.
However, Yao teaches wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises:
inputting the image to be segmented to an image segmentation model that has been pre-trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image (While Cheng teaches utilizing a model to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented (see claim 1), Yao teaches “the computer equipment performs iterative distillation training on the image segmentation model by using the sample image to obtain a trained image segmentation model with high image segmentation accuracy, and then can directly perform image segmentation processing on the image to be segmented by using the pre-trained image segmentation model” in para. [0183]. See also para. [0169] which teaches that the model is trained based on a sample image (sample segmented image, see para. [0170]), a sample segmentation result, “sample image semantic features”, and a segmentation marked image as shown in para. [0166]-[0169]. as shown in para. [0166]-[0169]. The sample pixel information as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the sample normal vector image as recited in the claim. Here, the guide model as described in the above cited sections is interpreted as equivalent to the claimed image segmentation model).
Cheng, Li, and Yao are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li) to incorporate the teachings of Yao and include “wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises: inputting the image to be segmented to an image segmentation model that has been pre-trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image”. The motivation for doing so would have been that, “by acquiring more parts of unlabeled sample images and a small part of labeled sample images, the resource consumption of labeled sample images is effectively reduced, and the accuracy of training the image segmentation model can be effectively improved. In the distillation training process, corresponding pixel errors and edge errors are respectively determined for unmarked sample images and marked sample images, so that the image segmentation model to be trained continuously learns knowledge in the guidance model according to the determined distillation errors, pixel errors and edge errors, and the accuracy of the image segmentation model can be effectively improved”, as suggested by Yao in para. [0136]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng and Li with Yao to obtain the invention specified in claim 2.
Regarding claim 3, Cheng, Li, and Yao teach the method according to claim 2, before the inputting the image to be segmented to the image segmentation model that has been pre-trained, further comprising:
with the sample segmented image as an input image to a big model that has been pre- established, and with the segmentation marked image and the sample normal vector image that are corresponding to the sample segmented image as expected output images of the big model, training the big model to obtain a teacher model (Yao teaches “inputting the sample image into the guide model to be trained, and performing image segmentation processing on the target object in the sample image through the guide model to be trained to obtain a sample segmentation result” as shown in para. [0160], and “after the computer equipment obtains the sample segmentation result, determining the pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel marked by the marking label. And simultaneously, determining an edge error according to the difference between the edge prediction result in the sample segmentation result and the edge label of the label” as shown in para. [0167], and finally “adjust[ing] the model parameters of the guidance model according to the pixel error and the edge error and continues iterative training until the training stopping condition is met to obtain the trained guidance model” as shown in para. [0168]. Here, the guidance model before training is interpreted as equivalent to the claimed big model, and the trained guidance model is interpreted as equivalent to the claimed teacher model. The marked information that the results of the guide model are compared to are interpreted as equivalent to the claimed sample information in the claim. The sample pixel information as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the sample normal vector image as recited in the claim); and
with the sample segmented image as an input image to a small model that has been pre- established (Yao teaches that “the image segmentation process is performed on the target object in the sample image 40 by the image segmentation model 42 to be trained, and a second segmentation result 4201 is obtained” in para. [0145], wherein the sample image 40 is interpreted as the sample segmented object and the segmentation model 42 is interpreted as equivalent to the claimed small model), and with a big model segmented image and a big model normal vector image that are corresponding to the sample segmented image and are output by the teacher model as expected outputs of the small model, training the small model to obtain the image segmentation model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model as shown in para. [0142] and “adjust[ing] the model parameters of the image segmentation model according to the distillation error and the edge error and continues distillation training until the training stopping condition is met to obtain the trained image segmentation model” as shown in para. [0147]. Here, the first segmentation result includes the claimed “big model segmented image” and “big model” pixel information, which can be combined with Cheng and Li’s teaching of the normal vector image to teach the claimed big model normal vector image).
Similar motivations as applied to claim 2 can be applied here.
Regarding claim 4, Cheng, Yao, and Li teach the method according to claim 3, wherein the training the big model to obtain the teacher model comprises:
inputting the sample segmented image to the big model that has been pre-established to obtain the big model segmented image and the big model normal vector image Yao teaches “inputting the sample image into the guide model to be trained, and performing image segmentation processing on the target object in the sample image through the guide model to be trained to obtain a sample segmentation result” as shown in para. [0160], Here, the guidance model before training is interpreted as equivalent to the claimed big model. The sample segmentation result is interpreted as equivalent to the claimed big model segmented image. Additionally, the “feature extraction is performed on the sample image through each network layer in the guidance model to be trained to extract sample image semantic features corresponding to the sample image, pixel points and contour edges belonging to a target object in the sample image are determined according to the sample image semantic features”, wherein the sample image semantic features as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the big model normal vector image as recited in the claim);
calculating a big model segmentation loss between the big model segmented image and the segmentation marked image corresponding to the sample segmented image, and calculating a big model normal vector loss between the big model normal vector image and the sample normal vector image corresponding to the sample segmented image (Yao teaches “after the computer equipment obtains the sample segmentation result, determining the pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel marked by the marking label. And simultaneously, determining an edge error according to the difference between the edge prediction result in the sample segmentation result and the edge label of the label” in para. [0167]. Here, the pixel error is interpreted as equivalent to the big model segmentation loss, and the edge error is interpreted as equivalent to the claimed big model normal vector loss); and
adjusting a model parameter of the big model based on the big model segmentation loss and the big model normal vector loss to obtain the teacher model (Yao teaches “adjusts the model parameters of the guidance model according to the pixel error and the edge error and continues iterative training until the training stopping condition is met to obtain the trained guidance model” in para. [0168]).
Similar motivation as applied to claim 2 can be applied here to claim 4.
Regarding claim 5, Cheng, Yao, and Li teach the method according to claim 4, wherein the calculating the big model segmentation loss between the big model segmented image and the segmentation marked image corresponding to the sample segmented image comprises:
calculating the big model segmentation loss between the big model segmented image and the segmentation marked image corresponding to the sample segmented image according to a binary cross entropy loss function (Yao teaches “the difference between the sample segmentation result and the label may also be measured by a loss function, for example, a cross entropy loss function” in para. [0168], wherein the cross-entropy loss produced a binarized pixel value error as shown in para. [0111]); or
calculating the big model segmentation loss between the big model segmented image and the segmentation marked image corresponding to the sample segmented image according to the binary cross entropy loss function and a regional mutual information loss function.
NOTE: Only one path above need be found in the prior art due to the “or” language in the claim. However, see also the following reference: Zhao et al. “Region Mutual Information Loss for Semantic Segmentation”, which could be combined with Cheng, Yao, and Li to teach the subject matter in the second pathway.
Similar motivation as applied to claim 2 can be applied here to claim 5.
Regarding claim 7, Cheng, Yao, and Li teach the method according to claim 3, wherein the training the small model comprises:
inputting the sample segmented image to the small model that has been pre-established as an input image to obtain a small model segmented image and a small model normal vector image (Yao teaches “the image segmentation process is performed on the target object in the sample image 40 by the image segmentation model 42 to be trained, and a second segmentation result 4201 is obtained” in para. [0145], wherein the second segmentation result 4201 includes a segmentation result (small model segmented image) and pixel information result (the image semantic features generated by the segmentation model as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector image as recited in the claim));
calculating a small model segmentation output loss based on the small model segmented image of the sample segmented image, the segmentation marked image, and the big model segmented image output by the teacher model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model (teacher model) as shown in para. [0142] and “the pixel error is determined from the difference between the pixel information in the second segmentation result 4201 and the pixel information in the first segmentation result 4104” in para. [0146], wherein the pixel error is interpreted as equivalent to the small model segmentation output loss);
calculating a small model normal vector output loss based on the small model normal vector image of the sample segmented image, the sample normal vector image, and the big model normal vector image output by the teacher model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model (teacher model) as shown in para. [0142] and “determining an edge error based on a difference between the edge predictor in the second segmentation result 4201 and the edge predictor in the first segmentation result 4104” in para. [0146], wherein the edge error is interpreted as equivalent to the small model normal vector output loss and the edge error determined based on the semantic level features as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector output loss as recited in the claim); and
adjusting a model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss to obtain the image segmentation model (Yao teaches “adjust[ing] the model parameters of the image segmentation model according to the distillation error and the edge error and continues distillation training until the training stopping condition is met to obtain the trained image segmentation model” as shown in para. [0147]).
Similar motivation as applied to claim 2 can be applied here to claim 7.
Regarding claim 9, Cheng, Li, and Yao teach the method according to claim 7, wherein the calculating the small model segmentation output loss based on the small model segmented image of the sample segmented image, the segmentation marked image, and the big model segmented image output by the teacher model comprises:
calculating a first small model segmentation loss between the small model segmented image of the sample segmented image and the segmentation marked image according to a binary cross entropy loss function (Yao teaches “the pixel error may include a binarization pixel value error of each pixel point and a pixel characteristic error corresponding to each pixel point or pixel area.” in para. [0117], wherein the cross-entropy loss produced a binarized pixel value error and this loss is calculated between the second predictrion result (small model segmented image) in the second segmentation result (small model segmented image) and the pixel information labeled by the label (segmentation marked image)), or the binary cross entropy loss function and a regional mutual information loss function;
calculating a second small model segmentation loss between the small model segmented image and the big model segmented image output by the teacher model according to a kullback-leibler divergence loss function (Yao teaches “the distillation error may be obtained by calculating a classification Loss using a KLLoss (relative entropy Loss) function, that is, comparing the classification Loss between the prediction class probability of each pixel in the sample image in the second prediction result and the prediction class probability of each pixel in the sample image in the first prediction result, and determining the distillation error between the image segmentation model to be trained and the guidance model” in para. [0110]. Here, the KL loss is interpreted as equivalent to the claimed kullback-leibler divergence loss function); and
determining the small model segmentation output loss based on the first small model segmentation loss and the second small model segmentation loss (Yao teaches “in the distillation training process, the distillation error, the pixel error and the edge error between the guidance model and the image segmentation model to be trained are respectively determined, so that the image segmentation model to be trained continuously learns knowledge in the guidance model according to the distillation error, the pixel error and the edge error, and the accuracy of the image segmentation model can be effectively improved” in para. [0120]. Here, the distillation error (second small model segmentation loss) may be determined using the kullback-leibler divergence loss function (see para. [0110] and the pixel error (first small model segmentation loss) may be determined using the binary cross entropy loss function (see para. [0117]), and both errors are used to determine the overall loss of the image segmentation model (small model)).
Regarding claim 17, Cheng and Li teach the electronic device according to claim 15.
While Cheng teaches obtaining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented (see claim 1), Cheng and Li fail to teach wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises: inputting the image to be segmented to an image segmentation model that has been pre- trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image.
However, Yao teaches wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises:
inputting the image to be segmented to an image segmentation model that has been pre-trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image (While Cheng teaches utilizing a model to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented (see claim 1), Yao teaches “the computer equipment performs iterative distillation training on the image segmentation model by using the sample image to obtain a trained image segmentation model with high image segmentation accuracy, and then can directly perform image segmentation processing on the image to be segmented by using the pre-trained image segmentation model” in para. [0183]. See also para. [0169] which teaches that the model is trained based on a sample image (sample segmented image, see para. [0170]), a sample segmentation result, “sample image semantic features”, and a segmentation marked image as shown in para. [0166]-[0169]. The sample pixel information as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the sample normal vector image as recited in the claim).
Cheng, Li, and Yao are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li) to incorporate the teachings of Yao and include “wherein the determining the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented comprises: inputting the image to be segmented to an image segmentation model that has been pre-trained to obtain the preliminarily segmented image and the target normal vector image that are corresponding to the image to be segmented, wherein the image segmentation model is trained based on a sample segmented image, a segmentation marked image corresponding to the sample segmented image, and a sample normal vector image corresponding to the sample segmented image”. The motivation for doing so would have been that, “by acquiring more parts of unlabeled sample images and a small part of labeled sample images, the resource consumption of labeled sample images is effectively reduced, and the accuracy of training the image segmentation model can be effectively improved. In the distillation training process, corresponding pixel errors and edge errors are respectively determined for unmarked sample images and marked sample images, so that the image segmentation model to be trained continuously learns knowledge in the guidance model according to the determined distillation errors, pixel errors and edge errors, and the accuracy of the image segmentation model can be effectively improved”, as suggested by Yao in para. [0136]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng and Li with Yao to obtain the invention specified in claim 17.
Regarding claim 18, Cheng, Li, and Yao teach the electronic device according to claim 17, before the inputting the image to be segmented to the image segmentation model that has been pre-trained, further comprising:
with the sample segmented image as an input image to a big model that has been pre- established, and with the segmentation marked image and the sample normal vector image that are corresponding to the sample segmented image as expected output images of the big model, training the big model to obtain a teacher model (Yao teaches “inputting the sample image into the guide model to be trained, and performing image segmentation processing on the target object in the sample image through the guide model to be trained to obtain a sample segmentation result” as shown in para. [0160], and “after the computer equipment obtains the sample segmentation result, determining the pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel marked by the marking label. And simultaneously, determining an edge error according to the difference between the edge prediction result in the sample segmentation result and the edge label of the label” as shown in para. [0167], and finally “adjust[ing] the model parameters of the guidance model according to the pixel error and the edge error and continues iterative training until the training stopping condition is met to obtain the trained guidance model” as shown in para. [0168]. Here, the guidance model before training is interpreted as equivalent to the claimed big model, and the trained guidance model is interpreted as equivalent to the claimed teacher model. The marked information that the results of the guide model are compared to are interpreted as equivalent to the claimed sample information in the claim. The sample pixel information as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the sample normal vector image as recited in the claim); and
with the sample segmented image as an input image to a small model that has been pre- established(Yao teaches that “the image segmentation process is performed on the target object in the sample image 40 by the image segmentation model 42 to be trained, and a second segmentation result 4201 is obtained” in para. [0145], wherein the sample image 40 is interpreted as the sample segmented object and the segmentation model 42 is interpreted as equivalent to the claimed small model), and with a big model segmented image and a big model normal vector image that are corresponding to the sample segmented image and are output by the teacher model as expected outputs of the small model, training the small model to obtain the image segmentation model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model as shown in para. [0142] and “adjust[ing] the model parameters of the image segmentation model according to the distillation error and the edge error and continues distillation training until the training stopping condition is met to obtain the trained image segmentation model” as shown in para. [0147]. Here, the first segmentation result includes the claimed “big model segmented image” and “big model” pixel information, which can be combined with Cheng and Li’s teaching of the normal vector image to teach the claimed big model normal vector image).
Similar motivations as applied to claim 17 can be applied here.
Regarding claim 19, Cheng, Yao, and Li teach the electronic device according to claim 18, wherein the training the big model to obtain the teacher model comprises:
inputting the sample segmented image to the big model that has been pre-established to obtain the big model segmented image and the big model normal vector image Yao teaches “inputting the sample image into the guide model to be trained, and performing image segmentation processing on the target object in the sample image through the guide model to be trained to obtain a sample segmentation result” as shown in para. [0160], Here, the guidance model before training is interpreted as equivalent to the claimed big model. The sample segmentation result is interpreted as equivalent to the claimed big model segmented image. Additionally, the “feature extraction is performed on the sample image through each network layer in the guidance model to be trained to extract sample image semantic features corresponding to the sample image, pixel points and contour edges belonging to a target object in the sample image are determined according to the sample image semantic features”, wherein the sample image semantic features as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the big model normal vector image as recited in the claim);
calculating a big model segmentation loss between the big model segmented image and the segmentation marked image corresponding to the sample segmented image, and calculating a big model normal vector loss between the big model normal vector image and the sample normal vector image corresponding to the sample segmented image (Yao teaches “after the computer equipment obtains the sample segmentation result, determining the pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel marked by the marking label. And simultaneously, determining an edge error according to the difference between the edge prediction result in the sample segmentation result and the edge label of the label” in para. [0167]. Here, the pixel error is interpreted as equivalent to the big model segmentation loss, and the edge error is interpreted as equivalent to the claimed big model normal vector loss); and
adjusting a model parameter of the big model based on the big model segmentation loss and the big model normal vector loss to obtain the teacher model (Yao teaches “adjusts the model parameters of the guidance model according to the pixel error and the edge error and continues iterative training until the training stopping condition is met to obtain the trained guidance model” in para. [0168]).
Similar motivation as applied to claim 17 can be applied here to claim 19.
Regarding claim 20, Cheng, Yao, and Li teach the electronic device according to claim 18, wherein the training the small model comprises:
inputting the sample segmented image to the small model that has been pre-established as an input image to obtain a small model segmented image and a small model normal vector image (Yao teaches “the image segmentation process is performed on the target object in the sample image 40 by the image segmentation model 42 to be trained, and a second segmentation result 4201 is obtained” in para. [0145], wherein the second segmentation result 4201 includes a segmentation result (small model segmented image) and pixel information result (the image semantic features generated by the segmentation model as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector image as recited in the claim));
calculating a small model segmentation output loss based on the small model segmented image of the sample segmented image, the segmentation marked image, and the big model segmented image output by the teacher model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model (teacher model) as shown in para. [0142] and “the pixel error is determined from the difference between the pixel information in the second segmentation result 4201 and the pixel information in the first segmentation result 4104” in para. [0146], wherein the pixel error is interpreted as equivalent to the small model segmentation output loss);
calculating a small model normal vector output loss based on the small model normal vector image of the sample segmented image, the sample normal vector image, and the big model normal vector image output by the teacher model (Yao teaches “determin[ing] the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104” in para. [0146], wherein the first segmentation result 4104 is determined by the guidance model (teacher model) as shown in para. [0142] and “determining an edge error based on a difference between the edge predictor in the second segmentation result 4201 and the edge predictor in the first segmentation result 4104” in para. [0146], wherein the edge error is interpreted as equivalent to the small model normal vector output loss and the edge error determined based on the semantic level features as taught by Yao can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector output loss as recited in the claim); and
adjusting a model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss to obtain the image segmentation model (Yao teaches “adjust[ing] the model parameters of the image segmentation model according to the distillation error and the edge error and continues distillation training until the training stopping condition is met to obtain the trained image segmentation model” as shown in para. [0147]).
Similar motivation as applied to claim 17 can be applied here to claim 20.
Claims 6 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li and Yao et al. (CN 113538480 A, see attached English translation for citations), hereinafter Yao and Gan et al. (CN 113538441 A, see attached English translation for citations), hereinafter Gan.
Regarding claim 6, Cheng, Yao, and Li teach the method according to claim 4, wherein the calculating the big model normal vector loss between the big model normal vector image and the sample normal vector image corresponding to the sample segmented image comprises:
calculating the big model normal vector loss between the big model normal vector image and the sample normal vector image corresponding to the sample segmented image according to a (Yao teaches “the difference between the sample segmentation result and the label may also be measured by a loss function, for example, a cross entropy loss function or other functions, an average absolute value loss function, a smooth average absolute error, or other functions may be selected as the loss function” in para. [0168]. Here, the sample image semantic features which make up the sample segmented image as taught by Yao in para. [0161]-[0162] can be combined with Cheng and Li’s teaching of the normal vector image to teach the big model normal vector image as recited in the claim).
Yao fails to specifically teach the above error function being a mean square error loss function.
However, Gan teaches a mean square error loss function (Gan teaches determining a loss “the non-uniformity loss function may be constructed based on KL divergence or mean square error loss” in para. [0161] according to a sample image in order to derive an accurate segmentation result).
Cheng, Li, Yao, and Gan are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li and Yao) to incorporate the teachings of Gan and include “a mean square error loss function”. The motivation for doing so would have been that, “in order to increase the accuracy of the lightweight segmentation model, a non-uniform loss function is introduced, so that the image segmentation capability of the lightweight segmentation model for different image transformations can be improved”, as suggested by Gan in para. [0160]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, and Yao with Gan to obtain the invention specified in claim 6.
Regarding claim 10, Cheng, Yao, and Li teach the method according to claim 7, wherein the calculating the small model normal vector output loss based on the small model normal vector image of the sample segmented image, the sample normal vector image, and the big model normal vector image output by the teacher model comprises:
calculating a first small model normal vector loss between the small model normal vector image of the sample segmented image and the sample normal vector image according to a (Yao teaches determining a loss produced a binarized pixel value error and this loss is calculated between the second prediction result (small model segmented image) in the second segmentation result (small model segmented image) and the pixel information labeled by the label (segmentation marked image) as shown in para. [0117]);
calculating a second small model normal vector loss between the small model normal vector image and the big model normal vector image output by the teacher model according to a kullback-leibler divergence loss function (Yao teaches “the distillation error may be obtained by calculating a classification Loss using a KLLoss (relative entropy Loss) function, that is, comparing the classification Loss between the prediction class probability of each pixel in the sample image in the second prediction result and the prediction class probability of each pixel in the sample image in the first prediction result, and determining the distillation error between the image segmentation model to be trained and the guidance model” in para. [0110]. Here, the KL loss is interpreted as equivalent to the claimed kullback-leibler divergence loss function); and
determining the small model normal vector output loss based on the first small model normal vector loss and the second small model normal vector loss (Yao teaches “in the distillation training process, the distillation error, the pixel error and the edge error between the guidance model and the image segmentation model to be trained are respectively determined, so that the image segmentation model to be trained continuously learns knowledge in the guidance model according to the distillation error, the pixel error and the edge error, and the accuracy of the image segmentation model can be effectively improved” in para. [0120]. Here, the distillation error (second small model segmentation loss) may be determined using the kullback-leibler divergence loss function (see para. [0110] and the pixel error (first small model segmentation loss) may be determined using a separate function (see para. [0117]), and both errors are used to determine the overall loss of the image segmentation model (small model)).
Yao fails to specifically teach the above error function being a mean square error loss function.
However, Gan teaches a mean square error loss function (Gan teaches determining a loss “the non-uniformity loss function may be constructed based on KL divergence or mean square error loss” in para. [0161] according to a sample image in order to derive an accurate segmentation result. Gan additionally teaches “constructing a total loss function according to the knowledge distillation loss function and the non-uniformity loss function; and updating the model parameters of the lightweight segmentation model to be trained according to the total loss function” in para. [0159]).
Cheng, Li, Yao, and Gan are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li and Yao) to incorporate the teachings of Gan and include “a mean square error loss function”. The motivation for doing so would have been that, “in order to increase the accuracy of the lightweight segmentation model, a non-uniform loss function is introduced, so that the image segmentation capability of the lightweight segmentation model for different image transformations can be improved”, as suggested by Gan in para. [0160]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, and Yao with Gan to obtain the invention specified in claim 10.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li, Yao et al. (CN 113538480 A, see attached English translation for citations), hereinafter Yao, Li et al. (CN 112465111 A, see attached English translation for citations), hereinafter Li ‘111, and Tang et al. (CN 114266777 A, see attached English translation for citations), hereinafter Tang.
Regarding claim 8, Cheng, Yao, and Li teach the method according to claim 7, further comprising:
the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises:
adjusting a model segmentation parameter of the small model based on the small model segmentation output loss (Yao teaches “further adjusting model parameters of the image segmentation model according to the distillation error and the edge error, and continuing distillation training until a training stopping condition is met to obtain the trained image segmentation model” in para. [0143]. Here, the distillation error is interpreted as equivalent to the claimed small model segmentation output loss)
adjusting a model normal vector parameter of the small model based on the small model normal vector output loss (Yao teaches “further adjusting model parameters of the image segmentation model according to the distillation error and the edge error, and continuing distillation training until a training stopping condition is met to obtain the trained image segmentation model” in para. [0143]. Here, the image segmentation model is interpreted as equivalent to the small model and the edge error, which is defined by the features of the image, can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector output loss as recited in the claim).
Cheng, Li, and Yao fail to teach inputting the small model segmented image output by the small model to a segmented image discriminator that has been pre-trained to obtain a segmentation discrimination result, and determining a segmentation discrimination loss based on the segmentation discrimination result and an expected discrimination result, wherein the segmented image discriminator is trained with the big model segmented image corresponding to the sample segmented image output by the teacher model as a real sample and the small model segmented image output by the small model as a fake sample; and the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises: adjusting a model segmentation parameter of the small model based on the segmentation discrimination loss.
However, Li ‘111 teaches inputting the small model segmented image output by the small model to a segmented image discriminator (Li ‘111 teaches “splicing the student network segmentation result and the original image, inputting the splicing result and the original image into a discriminator” in para. [0061], wherein the student network is interpreted as equivalent to the claimed small model and the GAN is trained to segment images as shown in para. [0011]), and
determining a segmentation discrimination loss based on the segmentation discrimination result and an expected discrimination result (Li ‘111 teaches “propagating forward to obtain a classification result, carrying out GANs loss to obtain loss4” in para. [0061]), wherein the segmented image discriminator is trained with the big model segmented image corresponding to the sample segmented image output by the teacher model as a real sample and the small model segmented image output by the small model as a fake sample (Li ‘111 teaches “rhe discriminator is a full convolution network, the characteristics are extracted through an average pooling layer in the middle, and finally two classification results are output to represent whether the image is true or false” in para. [0039], wherein “for the discriminator to be lost, the image label from the student network needs to be set to 0, and the image label from the teacher network needs to be set to 1” as shown in para. [0041]. Here, the discriminator is trained (see para. [0062]), by utilizing this process wherein the student network output (small model) is the fake sample (0) and the teacher network output (big model) is the real sample (1)); and
the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises:
adjusting a model segmentation parameter of the small model based on the segmentation discrimination loss (Li ‘111 teaches “splicing the student network segmentation result and the original image, inputting the splicing result and the original image into a discriminator, carrying out forward propagation to obtain a binary classification result, carrying out GANS loss to obtain a loss4, and finally carrying out optimization training by using an Adam optimizer in loss weighted fusion” in para. [0061], wherein loss4 is the segmentation discrimination loss, and the “the final student network loss L total is the weighted sum of the student network loss and GANs loss [wherein GANs loss is equivalent to loss4]” as shown in para. [0045]. Here, using an Adam optimizer to optimize the student network (small model) based on the GANs loss (segmentation discrimination loss) inherently involves adjusting the parameters of the network (model)).
Chen, Li, Yao, and Li ‘111 are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li and Yao) to incorporate the teachings of Li ‘111 and include “inputting the small model segmented image output by the small model to a segmented image discriminator that has been pre-trained to obtain a segmentation discrimination result, and determining a segmentation discrimination loss based on the segmentation discrimination result and an expected discrimination result, wherein the segmented image discriminator is trained with the big model segmented image corresponding to the sample segmented image output by the teacher model as a real sample and the small model segmented image output by the small model as a fake sample; and the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises: adjusting a model segmentation parameter of the small model based on the segmentation discrimination loss”. The motivation for doing so would have been that, “knowledge distillation and countermeasure training are combined, so that the easiness in judging of a discriminator in the countermeasure training can be improved, the robustness of the model is further improved, and the model of the segmentation method of the three-dimensional voxel image can be more easily put into production”, as suggested by Li ‘111 in para. [0012]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, and Yao with Li ‘111 to obtain the invention specified in the above claim limitations.
Cheng, Li, Yao, and Li ‘111 fail to specifically teach that the discriminator network is pre-trained to obtain a segmentation discrimination result.
However, Tang teaches a discriminator network that is pre-trained to obtain a segmentation discrimination result (Tang teaches “performing countermeasure training on the current segmentation model based on the fusion segmentation image and the pre-trained discrimination model” as shown in para. [0040]).
Chen, Li, Yao, Li ‘111, and Tang are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li, Yao, and Li ‘111) to incorporate the teachings of Tang and include “a discriminator network that is pre-trained to obtain a segmentation discrimination result”. The motivation for doing so would have been that, “a segmentation result image of the sample image based on different labels is obtained, so that the segmentation model can learn different label results under the same label, and the comprehensiveness of the output result of the segmentation model is improved” wherein the process of training the segmentation model includes “inputting the fusion segmentation image into a pre-trained discrimination model to obtain a discrimination result of the fusion image; further, the current segmentation model is trained based on the discrimination result”, as suggested by Tang in para. [0054] and [0069], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, Yao, and Li ‘111 with Tang to obtain the invention specified in claim 8.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li, Yao et al. (CN 113538480 A, see attached English translation for citations), hereinafter Yao, Zhou et al. (CN 113793341 A, see attached English translation for citations), hereinafter Zhou, and Tang et al. (CN 114266777 A, see attached English translation for citations), hereinafter Tang.
Regarding claim 11, Cheng, Yao, and Li teach the method according to claim 7, further comprising:
the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises:
adjusting a model segmentation parameter of the small model based on the small model segmentation output loss (Yao teaches “further adjusting model parameters of the image segmentation model according to the distillation error and the edge error, and continuing distillation training until a training stopping condition is met to obtain the trained image segmentation model” in para. [0143]. Here, the distillation error is interpreted as equivalent to the claimed small model segmentation output loss); and
adjusting a model normal vector parameter of the small model based on the small model normal vector output loss (Yao teaches “further adjusting model parameters of the image segmentation model according to the distillation error and the edge error, and continuing distillation training until a training stopping condition is met to obtain the trained image segmentation model” in para. [0143]. Here, the image segmentation model is interpreted as equivalent to the small model and the edge error, which is defined by the features of the image, can be combined with Cheng and Li’s teaching of the normal vector image to teach the small model normal vector output loss as recited in the claim).
Cheng, Li, and Yao fail to teach inputting the small model normal vector image output by the small model to a normal vector image discriminator that has been pre-trained to obtain a normal vector discrimination result, and determining a normal vector discrimination loss based on the normal vector discrimination result and an expected discrimination result, wherein the normal vector image discriminator is trained with the big model normal vector image corresponding to the sample segmented image output by the teacher model as a real sample and the small model normal vector image output by the small model as a fake sample; and the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises: adjusting a model normal vector parameter of the small model based on the normal vector discrimination loss.
However, Zhou teaches inputting the small model normal vector image output by the small model to a normal vector image discriminator (Zhou teaches “taking the feature graph output by the refining module as a true sample, taking the feature graph generated by the student network as a false sample, inputting the true sample and the false sample into a self-attention discriminator together for confrontation training” in para. [0012]. Here, the student network is interpreted as equivalent to the small model and the teacher network is interpreted as equivalent to the claimed big model. Additionally, the respective feature graphs of the student/teacher networks can be combined with the teaching of the normal vector images as taught by Chen in view of Li to teach the above limitation. Furthermore, the discriminator is trained using the feature graph of the teacher network (big model normal vector image corresponding to the sample segmented image) as a real sample and the feature graph of the student network (small model normal vector image corresponding to the sample segmented image) as a fake sample); and
the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises:
adjusting a model normal vector parameter of the small model based on the normal vector discrimination loss (Zhou teaches “inputting the true sample and the false sample into a self-attention discriminator together for confrontation training, optimizing the student network in the confrontation training” in para. [0012] wherein training the student model (small model) comprises “an effective optimization strategy [] on the basis of deep supervision loss ResNet” and “a self-attention discriminator which is composed of full convolution and can carry out confrontation training on the feature diagram obtained by refining the teacher network middle layer knowledge and the feature diagram of the student network middle layer, wherein two attention modules are inserted between the last three modules to capture the structural information” as shown in para. [0058]. Here, the confrontation training involves determining loss between the true/false student/teacher model outputs in order to optimize the student network. This loss is interpreted as equivalent to the claimed normal vector discrimination loss).
Chen, Li, Yao, and Zhou are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li and Yao) to incorporate the teachings of Zhou and include “inputting the small model normal vector image output by the small model to a normal vector image discriminator that has been pre-trained to obtain a normal vector discrimination result, and determining a normal vector discrimination loss based on the normal vector discrimination result and an expected discrimination result, wherein the normal vector image discriminator is trained with the big model normal vector image corresponding to the sample segmented image output by the teacher model as a real sample and the small model normal vector image output by the small model as a fake sample; and the adjusting the model parameter of the small model based on the small model segmentation output loss and the small model normal vector output loss comprises: adjusting a model normal vector parameter of the small model based on the normal vector discrimination loss”. The motivation for doing so would have been that, “the effect graph obtained by segmenting the student network by the knowledge distillation method has higher accuracy, which shows that the knowledge distillation method adopted by the embodiment can further improve the knowledge distillation effect and obtain a student model with higher accuracy”, as suggested by Zhou in para. [0079]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, and Yao with Zhou to obtain the invention specified in the above claim limitations.
Cheng, Li, Yao, and Zhou fail to specifically teach that the discriminator network is pre-trained to obtain a segmentation discrimination result.
However, Tang teaches a discriminator network that is pre-trained to obtain a segmentation discrimination result (Tang teaches “performing countermeasure training on the current segmentation model based on the fusion segmentation image and the pre-trained discrimination model” as shown in para. [0040]).
Chen, Li, Yao, Zhou, and Tang are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li, Yao, and Zhou) to incorporate the teachings of Tang and include “a discriminator network that is pre-trained to obtain a segmentation discrimination result”. The motivation for doing so would have been that, “a segmentation result image of the sample image based on different labels is obtained, so that the segmentation model can learn different label results under the same label, and the comprehensiveness of the output result of the segmentation model is improved” wherein the process of training the segmentation model includes “inputting the fusion segmentation image into a pre-trained discrimination model to obtain a discrimination result of the fusion image; further, the current segmentation model is trained based on the discrimination result”, as suggested by Tang in para. [0054] and [0069], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng, Li, Yao, and Zhou with Tang to obtain the invention specified in claim 11.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (CN 107871321 B, see attached English translation for citations), hereinafter Cheng, in view of Li et al. (U.S. Publication No. 2022/0180548 A1), hereinafter Li and Langlotz et al. (U.S. Publication No. 2013/0314442 A1), hereinafter Langlotz.
Regarding claim 13, Cheng and Li teach the method according to claim 1.
Chen and Li fail to teach after the performing image fusion on the preliminarily segmented image and the target normal vector image, further comprising: obtaining shooting angle information of an image shooting apparatus for shooting the image to be segmented, and adjusting the target segmented image based on the shooting angle information.
However, Langlotz teaches after the performing image fusion on the preliminarily segmented image and the target normal vector image, further comprising:
obtaining shooting angle information of an image shooting apparatus for shooting the image to be segmented (Langlotz teaches “the target camera captures a plurality of target image frames of the environment (226)”, wherein “the orientation of the target camera with respect to the environment may be determined while capturing the target video of the environment (228), e.g., using sensors 112, or the orientation may be determined using vision based techniques” in para. [0042]. Here, the determination of the orientation (shooting angle information) occurs before the image/frame is segmented, and at least one of the frames is interpreted as equivalent to the claimed image to be segmented. Additionally, Li teaches performing additional processing on the fused segmented image/target normal vector image as shown in Li in para. [0124]. As such, the process as taught by Langlotz can be combined with the teachings of the performing image fusion on the preliminarily segmented image and the target normal vector image as taught by Cheng in view of Li in order to teach a scenario in which Langlotz teaches determines the shooting information of a pre-processed image and then applies a transformation (as taught below by Langlotz) to the segmented fused image as taught by Cheng in view of Li), and
adjusting the target segmented image based on the shooting angle information (Langlotz teaches that “a transformation for the segmented video may be calculated using the source camera orientation for each frame of the segmented video and the orientation of the target camera for each frame of target video of the environment” in para. [0042]. Here, this transformation is interpreted as equivalent to the claimed adjustment and is applied to the segmented frames/video (target segmented image), but is based on the camera orientation of the unsegmented frames/video (image to be segmented)).
Cheng, Li, and Langlotz are all considered to be analogous to the claimed invention because they are in the same field of segmenting objects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Cheng (as modified by Li) to incorporate the teachings of Langlotz and include “after the performing image fusion on the preliminarily segmented image and the target normal vector image, further comprising: obtaining shooting angle information of an image shooting apparatus for shooting the image to be segmented, and adjusting the target segmented image based on the shooting angle information”. The motivation for doing so would have been that, “the segmented object may be displayed to the user so as to appear in the same position as when the source video was captured” and that “additional information such as the orientation of the source camera for each frame in the resulting segmented video of the object is also determined and stored to assist in registering the extracted object into a subsequently acquired target video of the same or different environment”, as suggested by Langlotz in para. [0042] and para. [0043], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Cheng and Li with Langlotz to obtain the invention specified in claim 13.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang (CN 111862124 A) teaches adjusting an image to be segmented based on a shooting angle.
Zhao et al. (“Region Mutual Information Loss for Semantic Segmentation”) additionally teaches region mutual information.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights