DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 01/17/2025 and 07/012026 is being considered by the examiner.
Claim Objections
Claims 1-3, 10-11, and 19-20 are objected to because of the following informalities:
In claim 1, line 4, the term “training the target semantic segmentation model” should be changed to “training a target semantic segmentation model” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 2, line 9, the term “wherein the feature value represents” should be changed to “wherein the at least one feature value represents” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 3, line 1, the term “wherein the feature value comprises” should be changed to “wherein the at least one feature value comprises” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 10, line 9-10, the term “training the target semantic segmentation model” should be changed to “training a target semantic segmentation model” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 11, line 7-8, the term “training the target semantic segmentation model” should be changed to “training a target semantic segmentation model” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 19, line 9-10, the term “wherein the feature value represents” should be changed to “wherein the at least one feature value represents” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
In claim 20, line 1, the term “wherein the feature value comprises” should be changed to “wherein the at least one feature value comprises” in order to avoid an insufficient antecedent issue and prevent a rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function.
Claims 2, 3, 6, 10, 20, and 23 recite limitations that use words like “means” (or “step”) or similar terms with functional language but do not invoke 35 U.S.C. 112(f):
Claim 3; recites the limitation, “the information entropy evaluation value is configured to……,” [Line 4].
Claim 3; recites the limitation, “the difficulty evaluation value is configured to……,” [Line 6].
Claim 3; recites the limitation, “the diversity evaluation value is configured to……,” [Line 8].
Claim 3; recites the limitation, “the consistency evaluation value is configured to……,” [Line 10].
Claim 10; recites the limitation, “the processor is configured to……,” [Line 4].
Claim 20; recites the limitation, “the information entropy evaluation value is configured to……,” [Line 4].
Claim 20; recites the limitation, “the difficulty evaluation value is configured to……,” [Line 6].
Claim 20; recites the limitation, “the diversity evaluation value is configured to……,” [Line 8].
Claim 20; recites the limitation, “the consistency evaluation value is configured to……,” [Line 10].
Claim 2; recites the limitation, “output by the first codec network …..” [Line 6].
Claim 2; recites the limitation, “output by the second codec network …..” [Line 7].
Claim 6; recites the limitation, “output by the first codec network …..” [Line 7].
Claim 6; recites the limitation, “output by the second codec network …..” [Line 10].
Claim 19; recites the limitation, “output by the first codec network …..” [Line 6].
Claim 19; recites the limitation, “output by the second codec network …..” [Line 7].
Claim 23; recites the limitation, “output by the first codec network …..” [Line 8].
Claim 23; recites the limitation, “output by the second codec network …..” [Line 11].
Such claim limitation(s) is/are:
“the information entropy evaluation value……,” has a structure associated with it a value/number.
“the difficulty evaluation value……,” has a structure associated with it a value/number.
“the diversity evaluation value……,” has a structure associated with it a value/number.
“the consistency evaluation value……,” has a structure associated with it a value/number.
“the processor……,” has a structure associated with it a processor.
“codec network…….” has a structure associated with it a network.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 10-11, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (US 20210166150 A1), hereinafter referenced as WANG, in view of ZHANG et al. (US 20210056417 A1), hereinafter referenced as ZHANG.
Regarding claim 1, WANG explicitly teaches an image semantic segmentation model optimization method (Fig. 2. Paragraph [0046]-WANG discloses FIG. 2 is a flow diagram illustrating a process 200 for integrating bottom-up segmentation into a semi-supervised machine learning model, in accordance with embodiments of the present disclosure. Further in paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image.), comprising:
acquiring first unlabeled data (Fig. 1, #120 called unlabeled dataset. Paragraph [0031]-WANG discloses the unlabeled dataset 120 is a set of data used by the image segmentation system 100 to apply to the machine learning model 130 to predict the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 (wherein applying the unlabeled data is acquiring first unlabeled data).), and
evaluating the first unlabeled data based on a pre-trained target semantic segmentation model (Fig. 1, #130 called machine learning model. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a pre-trained target semantic segmentation model).) to obtain an evaluation value corresponding to the first unlabeled data (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).),
wherein the evaluation value represents effectiveness of training the target semantic segmentation model (Fig. 1. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a target semantic segmentation model).) with the first unlabeled data (Fig. 2. Paragraph [0051]-WANG discloses a bottom-up segmentation grouping rule can be selected from the bottom-up grouping rules 138 to capture how well the machine learning model 1340 predicted segmentation agrees with the image under the selected bottom-up semantic segmentation grouping rule. Further in paragraph [0053]-WANG discloses the process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein how well the machine learning model agrees with the bottom-up image is the evaluation value that represents the effectiveness of training).);
optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain an optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).).
WANG fails to explicitly teach determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
However, ZHANG explicitly teaches determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0030]-ZHANG discloses the sample selector 150 selects a threshold number of unlabeled training samples 112U.sub.T based on their respective inconsistency values 142 to form a current set of unlabeled training samples 112U.sub.T (wherein the selected unlabeled training samples 112U.sub.T are target unlabeled data).), and
generating target labeled data corresponding to the target unlabeled data (Fig. 3A. Paragraph [0030]-ZHANG discloses the ground truth labels 132G are labels that are empirically determined by another source. In some implementations, an oracle 160 determines the ground truth labels 132G of the unlabeled training samples 112U.sub.T (wherein the unlabeled training samples 112U.sub.T are target unlabeled data).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 5, WANG in view of ZHANG explicitly teach the method according to claim 1,
WANG further explicitly teaches wherein the optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain the optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).), comprises:
performing semi-supervised training on the image semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation (wherein the machine learning model 130 is the image semantic segmentation model).) with the target labeled data and the second unlabeled data (Fig. 2. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100 (wherein the top-down pseudo labeled dataset is the target labeled data and the bottom-up evaluation results is the second unlabeled data).) to obtain the optimized semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein the retrained machine learning model is the optimized semantic segmentation model).).
WANG fails to explicitly teach acquiring second unlabeled data with a same amount as the target labeled data; and.
However, ZHANG explicitly teaches acquiring second unlabeled data with a same amount as the target labeled data (Fig. 2. Paragraph [0031]-ZHANG discloses the oracle 160, in response to receiving the unlabeled training samples 112U.sub.T, determines or otherwise obtains the associated ground truth label 132G for each unlabeled training sample 112U r. The unlabeled training samples 112U.sub.T, combined with the ground truth labels 132G, form labeled training samples 112L and may be stored with other labeled training samples 112L (e.g., the labeled training samples 112L that the model trainer 110 used to initially train the target model 130). That is, the model trainer 110 may select a current set of labeled training samples 112L that includes the selected unlabeled training samples 110U.sub.T paired with the corresponding ground truth labels 132G.); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of acquiring second unlabeled data with a same amount as the target labeled data; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having acquiring second unlabeled data with a same amount as the target labeled data; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 10, WANG explicitly teaches an electronic device (Fig. 3. Paragraph [0058]-WANG discloses the computer system 300 may be a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface, but receives requests from other computer systems (clients). Further, in some embodiments, the computer system 300 may be implemented as a desktop computer, portable computer, laptop or notebook computer, tablet computer, pocket computer, telephone, smart phone, network switches or routers, or any other appropriate type of electronic device.),
comprising a processor (Fig. 3, #302 called processor. Paragraph [0055]-WANG discloses the computer system 300 may contain one or more general-purpose programmable central processing units (CPUs) 302-1, 302-2, 302-3, and 302-N, herein generically referred to as the processor 302.) and a memory (Fig. 3, #304 called memory. Paragraph [0056]-WANG discloses the memory 304 may include computer system readable media in the form of volatile memory, such as random-access memory (RAM) 322 or cache memory 324.) in communication connection with the processor (Fig. 3. Paragraph [0055]-WANG discloses each processor 301 may execute instructions stored in the memory 304 and may include one or more levels of on-board cache. Further in paragraph [0057]-WANG discloses although the memory bus 303 is shown in FIG. 3 as a single bus structure providing a direct communication path among the processors 302, the memory 304, and the I/O bus interface 310, the memory bus 303 may, in some embodiments, include multiple different buses or communication paths, which may be arranged in any of various forms, such as point-to-point links in hierarchical, star or web configurations, multiple hierarchical buses, parallel and redundant paths, or any other appropriate type of configuration.),
wherein the memory stores computer-executable instructions (Fig. 3. Paragraph [0055]-WANG discloses each processor 301 may execute instructions stored in the memory 304 and may include one or more levels of on-board cache.); and
the processor is configured to execute the computer-executable instructions stored in the memory (Fig. 3. Paragraph [0055]-WANG discloses each processor 301 may execute instructions stored in the memory 304 and may include one or more levels of on-board cache.) to implement an image semantic segmentation model optimization (Fig. 2. Paragraph [0053]-WANG discloses as a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training.), which comprises:
acquiring first unlabeled data (Fig. 1, #120 called unlabeled dataset. Paragraph [0031]-WANG discloses the unlabeled dataset 120 is a set of data used by the image segmentation system 100 to apply to the machine learning model 130 to predict the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 (wherein applying the unlabeled data is acquiring first unlabeled data).), and
evaluating the first unlabeled data based on a pre-trained target semantic segmentation model (Fig. 1, #130 called machine learning model. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a pre-trained target semantic segmentation model).) to obtain an evaluation value corresponding to the first unlabeled data (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).),
wherein the evaluation value represents effectiveness of training the target semantic segmentation model (Fig. 1. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a target semantic segmentation model).) with the first unlabeled data (Fig. 2. Paragraph [0051]-WANG discloses a bottom-up segmentation grouping rule can be selected from the bottom-up grouping rules 138 to capture how well the machine learning model 1340 predicted segmentation agrees with the image under the selected bottom-up semantic segmentation grouping rule. Further in paragraph [0053]-WANG discloses the process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein how well the machine learning model agrees with the bottom-up image is the evaluation value that represents the effectiveness of training).);
optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain an optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).).
WANG fails to explicitly teach determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
However, ZHANG explicitly teaches determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0030]-ZHANG discloses the sample selector 150 selects a threshold number of unlabeled training samples 112U.sub.T based on their respective inconsistency values 142 to form a current set of unlabeled training samples 112U.sub.T (wherein the selected unlabeled training samples 112U.sub.T are target unlabeled data).), and
generating target labeled data corresponding to the target unlabeled data (Fig. 3A. Paragraph [0030]-ZHANG discloses the ground truth labels 132G are labels that are empirically determined by another source. In some implementations, an oracle 160 determines the ground truth labels 132G of the unlabeled training samples 112U.sub.T (wherein the unlabeled training samples 112U.sub.T are target unlabeled data).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an electronic device, comprising a processor and a memory in communication connection with the processor, wherein the memory stores computer-executable instructions; and the processor is configured to execute the computer-executable instructions stored in the memory to implement an image semantic segmentation model optimization method, which comprises acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
Wherein having WANG’s device for a semi-supervised image semantic segmentation method having determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
The motivation behind the modification would have been to obtain a device that processes a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 11, WANG explicitly teaches a non-transitory computer-readable storage medium (Fig. 3, #304 called memory. Paragraph [0086]-WANG discloses the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Further in paragraph [0086]-WANG discloses a computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.), storing computer-executable instructions (Fig. 3, #304 called memory. Paragraph [0086]-WANG discloses the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.), wherein a processor (Fig. 3, #302 called processor. Paragraph [0055]-WANG discloses the computer system 300 may contain one or more general-purpose programmable central processing units (CPUs) 302-1, 302-2, 302-3, and 302-N, herein generically referred to as the processor 302.), when executing the computer-executable instructions (Fig. 3. Paragraph [0055]-WANG discloses each processor 301 may execute instructions stored in the memory 304 and may include one or more levels of on-board cache.), implements an image semantic segmentation model optimization method (Fig. 2. Paragraph [0053]-WANG discloses as a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training.), which comprises:
acquiring first unlabeled data (Fig. 1, #120 called unlabeled dataset. Paragraph [0031]-WANG discloses the unlabeled dataset 120 is a set of data used by the image segmentation system 100 to apply to the machine learning model 130 to predict the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 (wherein applying the unlabeled data is acquiring first unlabeled data).), and
evaluating the first unlabeled data based on a pre-trained target semantic segmentation model (Fig. 1, #130 called machine learning model. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a pre-trained target semantic segmentation model).) to obtain an evaluation value corresponding to the first unlabeled data (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).),
wherein the evaluation value represents effectiveness of training the target semantic segmentation model (Fig. 1. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a target semantic segmentation model).) with the first unlabeled data (Fig. 2. Paragraph [0051]-WANG discloses a bottom-up segmentation grouping rule can be selected from the bottom-up grouping rules 138 to capture how well the machine learning model 1340 predicted segmentation agrees with the image under the selected bottom-up semantic segmentation grouping rule. Further in paragraph [0053]-WANG discloses the process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein how well the machine learning model agrees with the bottom-up image is the evaluation value that represents the effectiveness of training).);
optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain an optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).).
WANG fails to explicitly teach determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
However, ZHANG explicitly teaches determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0030]-ZHANG discloses the sample selector 150 selects a threshold number of unlabeled training samples 112U.sub.T based on their respective inconsistency values 142 to form a current set of unlabeled training samples 112U.sub.T (wherein the selected unlabeled training samples 112U.sub.T are target unlabeled data).), and
generating target labeled data corresponding to the target unlabeled data (Fig. 3A. Paragraph [0030]-ZHANG discloses the ground truth labels 132G are labels that are empirically determined by another source. In some implementations, an oracle 160 determines the ground truth labels 132G of the unlabeled training samples 112U.sub.T (wherein the unlabeled training samples 112U.sub.T are target unlabeled data).); and.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of a non-transitory computer-readable storage medium, storing computer-executable instructions, wherein a processor, when executing the computer-executable instructions, implements an image semantic segmentation model optimization method, which comprises: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having determining target unlabeled data according to the evaluation value corresponding to the first unlabeled data, and generating target labeled data corresponding to the target unlabeled data; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 22, WANG in view of ZHANG explicitly teach the electronic device according to claim 10,
WANG further explicitly teaches wherein the optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain the optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).), comprises:
performing semi-supervised training on the image semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation (wherein the machine learning model 130 is the image semantic segmentation model).) with the target labeled data and the second unlabeled data (Fig. 2. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100 (wherein the top-down pseudo labeled dataset is the target labeled data and the bottom-up evaluation results is the second unlabeled data).) to obtain the optimized semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein the retrained machine learning model is the optimized semantic segmentation model).).
WANG fails to explicitly teach acquiring second unlabeled data with a same amount as the target labeled data; and.
However, ZHANG explicitly teaches acquiring second unlabeled data with a same amount as the target labeled data (Fig. 2. Paragraph [0031]-ZHANG discloses the oracle 160, in response to receiving the unlabeled training samples 112U.sub.T, determines or otherwise obtains the associated ground truth label 132G for each unlabeled training sample 112U r. The unlabeled training samples 112U.sub.T, combined with the ground truth labels 132G, form labeled training samples 112L and may be stored with other labeled training samples 112L (e.g., the labeled training samples 112L that the model trainer 110 used to initially train the target model 130). That is, the model trainer 110 may select a current set of labeled training samples 112L that includes the selected unlabeled training samples 110U.sub.T paired with the corresponding ground truth labels 132G.); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an electronic device, comprising a processor and a memory in communication connection with the processor, wherein the memory stores computer-executable instructions; and the processor is configured to execute the computer-executable instructions stored in the memory to implement an image semantic segmentation model optimization method, which comprises acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of acquiring second unlabeled data with a same amount as the target labeled data; and.
Wherein having WANG’s device for semi-supervised image semantic segmentation method having acquiring second unlabeled data with a same amount as the target labeled data; and.
The motivation behind the modification would have been to obtain a device that processes a semi-supervised image semantic segmentation method that reduce the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Claims 2, 15, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (US 20210166150 A1), hereinafter referenced as WANG, in view of ZHANG et al. (US 20210056417 A1), hereinafter referenced as ZHANG, and further in view of ZHI et al. (US 20210382497 A1), hereinafter referenced as ZHI.
Regarding claim 2, WANG in view of ZHANG explicitly teach the method according to claim 1,
WANG further explicitly teaches the evaluating the first unlabeled data based on the pre-trained target semantic segmentation (Fig. 1, #130 called machine learning model. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a pre-trained target semantic segmentation model).) model to obtain the evaluation value corresponding to the first unlabeled data (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).), comprises:
processing the first segmentation result image and the second segmentation result image based on a preset sample evaluation model to obtain at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein the ground truth labels are the first segmentation result image, the model predicted results are the second segmentation result image, L.sub.sup is the feature value, and equation 1 is a preset sample evaluation model).),
wherein the feature value represents an evaluation result of the first unlabeled data in a corresponding evaluation dimension (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup is the feature value).); and
performing weighting fusion on the at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup is the feature value) to obtain the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the feature value is weighted in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).).
Although WANG explicitly teaches processing the first unlabeled data, WANG in view of ZHANG fail to explicitly teach wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively.
However, ZHI explicitly teaches wherein the target semantic segmentation model (Fig. 12, #1200 called latent representation prediction engine. Paragraph [0150]-ZHI discloses the latent representation prediction engine 1200 receives image data 1202. The image data 1202 in this case is training data, which is used to train the latent representation prediction engine 1200 to predict a semantic segmentation of an input image (wherein the latent representation prediction engine is the target semantic segmentation model).) comprises a first codec network and a second codec network (Fig. 12. Paragraph [0156]-ZHI discloses the first encoder 1214 and the first decoder 1224 in this example correspond to a first autoencoder, which is to be trained to autoencode a semantic segmentation of an input image. The second encoder 1216 and the second decoder 1226 in this example correspond to a second autoencoder, which is to be trained to autoencode a depth map of an input image. Please see annotated Fig. 12 below.), and
processing the first unlabeled data based on the first codec network and the second codec network (Fig. 12, #1202 called image data. Paragraph [0150]-ZHI discloses the image data 1202 in this case is training data, which is used to train the latent representation prediction engine 1200 to predict a semantic segmentation of an input image. In this example, the image data 1202 includes image data representative of the input image, e.g. as a 2D array of pixel values (such as pixel intensity values). For example, the image may be a colour image (wherein the image data is the first unlabeled data).) to obtain a first segmentation result image output by the first codec network (Fig. 12. Paragraph [0157]-ZHI discloses the first encoder 1214 and the first decoder 1224 may be trained to perform variational autoencoding of an input semantic segmentation of an input image. Further in paragraph [0158]-ZHI discloses the first decoder 1224 is arranged to output a predicted semantic segmentation 1228 of an input image (wherein a predicted semantic segmentation is a first segmentation result image).) and a second segmentation result image output by the second codec network (Fig. 12. Paragraph [0158]-ZHI discloses the second decoder 1226 is arranged to output a predicted depth map 1230 of an input image. The predicted semantic segmentation 1228 and the predicted depth map 1230 may be a normalised semantic segmentation or depth map. Normalisation may be performed by the decoder system 1222 (e.g. by the first decoder 1224 and/or the second decoder 1226) or by another component (wherein the depth map is a second segmentation result image).), respectively;
PNG
media_image3.png
515
491
media_image3.png
Greyscale
Annotated diagram of ZHI’s Fig. 12 illustrating two codec networks.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG in view of ZHANG of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHI of wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively.
Wherein having WANG’s semi-supervised image semantic segmentation method having wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHI relate to processing image data to perform image semantic segmentation, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHI there is still a desire for efficient representations of scenes, which provide information on what is visible in a scene. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHI et al. (US 20210382497 A1), Paragraph [0009].
Regarding claim 15, WANG in view of ZHANG and further in view of ZHI explicitly teach the method according to claim 2,
WANG further explicitly teaches wherein the optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain the optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).), comprises:
performing semi-supervised training on the image semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation (wherein the machine learning model 130 is the image semantic segmentation model).) with the target labeled data and the second unlabeled data (Fig. 2. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100 (wherein the top-down pseudo labeled dataset is the target labeled data and the bottom-up evaluation results is the second unlabeled data).) to obtain the optimized semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein the retrained machine learning model is the optimized semantic segmentation model).).
WANG fails to explicitly teach acquiring second unlabeled data with a same amount as the target labeled data; and.
However, ZHANG explicitly teaches acquiring second unlabeled data with a same amount as the target labeled data (Fig. 2. Paragraph [0031]-ZHANG discloses the oracle 160, in response to receiving the unlabeled training samples 112U.sub.T, determines or otherwise obtains the associated ground truth label 132G for each unlabeled training sample 112U r. The unlabeled training samples 112U.sub.T, combined with the ground truth labels 132G, form labeled training samples 112L and may be stored with other labeled training samples 112L (e.g., the labeled training samples 112L that the model trainer 110 used to initially train the target model 130). That is, the model trainer 110 may select a current set of labeled training samples 112L that includes the selected unlabeled training samples 110U.sub.T paired with the corresponding ground truth labels 132G.); and.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of acquiring second unlabeled data with a same amount as the target labeled data; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having acquiring second unlabeled data with a same amount as the target labeled data; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 19, WANG in view of ZHANG explicitly teaches the electronic device according to claim 10,
WANG further explicitly teaches the evaluating the first unlabeled data based on the pre-trained target semantic segmentation model (Fig. 1, #130 called machine learning model. Paragraph [0047]-WANG discloses once a machine learning algorithm is selected, the machine learning model 130 is trained to perform semantic segmentation on an image (wherein the machine learning model trained to perform semantic segmentation is a pre-trained target semantic segmentation model).) model to obtain the evaluation value corresponding to the first unlabeled data to obtain the evaluation value corresponding to the first unlabeled data (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).), comprises:
processing the first segmentation result image and the second segmentation result image based on a preset sample evaluation model to obtain at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein the ground truth labels are the first segmentation result image, the model predicted results are the second segmentation result image, L.sub.sup is the feature value, and equation 1 is a preset sample evaluation model).),
wherein the feature value represents an evaluation result of the first unlabeled data in a corresponding evaluation dimension (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup is the feature value).); and
performing weighting fusion on the at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup is the feature value).) to obtain the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the feature value is weighted in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).).
Although WANG explicitly teaches processing the first unlabeled data, WANG in view of ZHANG fail to explicitly teach wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively;
However, ZHI explicitly teaches wherein the target semantic segmentation model (Fig. 12, #1200 called latent representation prediction engine. Paragraph [0150]-ZHI discloses the latent representation prediction engine 1200 receives image data 1202. The image data 1202 in this case is training data, which is used to train the latent representation prediction engine 1200 to predict a semantic segmentation of an input image (wherein the latent representation prediction engine is the target semantic segmentation model).) comprises a first codec network and a second codec network (Fig. 12. Paragraph [0156]-ZHI discloses the first encoder 1214 and the first decoder 1224 in this example correspond to a first autoencoder, which is to be trained to autoencode a semantic segmentation of an input image. The second encoder 1216 and the second decoder 1226 in this example correspond to a second autoencoder, which is to be trained to autoencode a depth map of an input image. Please see annotated Fig. 12 below.), and
processing the first unlabeled data based on the first codec network and the second codec network (Fig. 12, #1202 called image data. Paragraph [0150]-ZHI discloses the image data 1202 in this case is training data, which is used to train the latent representation prediction engine 1200 to predict a semantic segmentation of an input image. In this example, the image data 1202 includes image data representative of the input image, e.g. as a 2D array of pixel values (such as pixel intensity values). For example, the image may be a colour image (wherein the image data is the first unlabeled data).) to obtain a first segmentation result image output by the first codec network (Fig. 12. Paragraph [0157]-ZHI discloses the first encoder 1214 and the first decoder 1224 may be trained to perform variational autoencoding of an input semantic segmentation of an input image. Further in paragraph [0158]-ZHI discloses the first decoder 1224 is arranged to output a predicted semantic segmentation 1228 of an input image (wherein a predicted semantic segmentation is a first segmentation result image).) and a second segmentation result image output by the second codec network (Fig. 12. Paragraph [0158]-ZHI discloses the second decoder 1226 is arranged to output a predicted depth map 1230 of an input image. The predicted semantic segmentation 1228 and the predicted depth map 1230 may be a normalised semantic segmentation or depth map. Normalisation may be performed by the decoder system 1222 (e.g. by the first decoder 1224 and/or the second decoder 1226) or by another component (wherein the depth map is a second segmentation result image).), respectively;
PNG
media_image3.png
515
491
media_image3.png
Greyscale
Annotated diagram of ZHI’s Fig. 12 illustrating two codec networks.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG in view of ZHANG of an electronic device, comprising a processor and a memory in communication connection with the processor, wherein the memory stores computer-executable instructions; and the processor is configured to execute the computer-executable instructions stored in the memory to implement an image semantic segmentation model optimization method, which comprises acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHI of wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively.
Wherein having WANG’s device for semi-supervised image semantic segmentation method having wherein the target semantic segmentation model comprises a first codec network and a second codec network, and processing the first unlabeled data based on the first codec network and the second codec network to obtain a first segmentation result image output by the first codec network and a second segmentation result image output by the second codec network, respectively.
The motivation behind the modification would have been to obtain a device that processes a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHI relate to processing image data to perform image semantic segmentation, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHI there is still a desire for efficient representations of scenes, which provide information on what is visible in a scene. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHI et al. (US 20210382497 A1), Paragraph [0009].
Claims 4, 17, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over WANG et al. (US 20210166150 A1), hereinafter referenced as WANG, in view of ZHANG et al. (US 20210056417 A1), hereinafter referenced as ZHANG, and further in view of ZHI et al. (US 20210382497 A1), hereinafter referenced as ZHI, and further in view of ZHAO et al. (US 20210241097 A1), hereinafter referenced as ZHAO.
Regarding claim 4, WANG in view of ZHANG and further in view of ZHI explicitly teach the method according to claim 2,
WANG further explicitly teaches wherein the performing weighting fusion on the at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) is the feature value and the weighted cross entropy function performs weighting fusion).) to obtain the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the feature value is weighted in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).), comprises:
calculating a weighted sum of the respective feature values according to the respective weighting coefficients to obtain the evaluation value (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).) corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the weighted sum is performed in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).).
WANG in view of ZHANG and further in view of ZHI fail to explicitly teach acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and
However, ZHAO explicitly teaches acquiring respective weighting coefficients corresponding to respective feature values (Fig. 4A. Paragraph [0070]-ZHAO discloses the apparatus 400 includes a loss determination unit 401 configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an updating unit 402 configured to perform an updating operation for parameters of the neural network model based on the loss data (wherein the parameters of the neural network updated based on the loss data of the extracted features are weighting coefficients corresponding to respective feature values).),
wherein the weighting coefficients (Fig. 4A. Paragraph [0070]-ZHAO discloses the apparatus 400 includes a loss determination unit 401 configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an updating unit 402 configured to perform an updating operation for parameters of the neural network model based on the loss data (wherein the parameters of the neural network updated based on the loss data of the extracted features are weighting coefficients corresponding to respective feature values).) are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model (Fig. 1. Paragraph [0084]-ZHAO discloses it is possible to set two initial values for one parameter of a weight function, and then use each parameter value to perform iterative loss data determination and updating operations, and after one round of training for each parameter completes, a parameter value that leads to a better training result (for example, the loss data caused by the trained model) is chosen, and two parameter values around the chosen parameter value are set as parameters for the weight function used in the next round of training operation. Such process is repeated until the predetermined number of adjustments is reached or the training result is no longer better. Further in paragraph [0047]-ZHAO discloses the obtained classification probabilities are compared with real situation values 0, 1, 0, . . . , 0 (where 1 indicates the true value) to determine the difference therebetween, such as cross entropy, as the loss data.); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG in view of ZHANG and further in view of ZHI of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHAO of acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHAO relate to training image processing machine learning models with loss functions, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHAO there is a need for an improved technique to improve the training of object recognition models. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHAO et al. (US 20210241097 A1), Paragraph [0006].
Regarding claim 17, WANG in view of ZHANG and further in view of ZHI and further in view of ZHAO explicitly teach the method according to claim 4,
WANG further explicitly teaches wherein the optimizing the target semantic segmentation model based on the target labeled data (Fig. 1. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100.) to obtain the optimized semantic segmentation model (Fig. 2, #250 called retraining machine learning model. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130 (wherein retraining with the combined dataset is optimizing the target semantic segmentation model).), comprises:
performing semi-supervised training on the image semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation (wherein the machine learning model 130 is the image semantic segmentation model).) with the target labeled data and the second unlabeled data (Fig. 2. Paragraph [0043]-WANG discloses the combined dataset 160 is a combination of the top-down pseudo labeled dataset 140 and the bottom-up evaluation results 150 combined for retraining the image segmentation system 100 (wherein the top-down pseudo labeled dataset is the target labeled data and the bottom-up evaluation results is the second unlabeled data).) to obtain the optimized semantic segmentation model (Fig. 2. Paragraph [0053]-WANG discloses once combined, the combined dataset 160 is used to retrain the machine learning model 130. This is illustrated at step 250. The retraining process can be used to improve the predicted probability of the segmentation performed by the machine learning model 130. As a result, the process 200 produces a semi-supervised method for semantic image segmentation that integrates prior knowledge developed in bottom-up unsupervised segmentation with top-down supervised segmentation. The process 200 trains the machine learning model 130 with unlabeled images by evaluating how well the machine learning model 130 performs on unlabeled images and applies the evaluation results to improve training (wherein the retrained machine learning model is the optimized semantic segmentation model).).
WANG fails to explicitly teach acquiring second unlabeled data with a same amount as the target labeled data; and.
However, ZHANG explicitly teaches acquiring second unlabeled data with a same amount as the target labeled data (Fig. 2. Paragraph [0031]-ZHANG discloses the oracle 160, in response to receiving the unlabeled training samples 112U.sub.T, determines or otherwise obtains the associated ground truth label 132G for each unlabeled training sample 112U r. The unlabeled training samples 112U.sub.T, combined with the ground truth labels 132G, form labeled training samples 112L and may be stored with other labeled training samples 112L (e.g., the labeled training samples 112L that the model trainer 110 used to initially train the target model 130). That is, the model trainer 110 may select a current set of labeled training samples 112L that includes the selected unlabeled training samples 110U.sub.T paired with the corresponding ground truth labels 132G.); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG of an image semantic segmentation model optimization method, comprising: acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHANG of acquiring second unlabeled data with a same amount as the target labeled data; and.
Wherein having WANG’s semi-supervised image semantic segmentation method having acquiring second unlabeled data with a same amount as the target labeled data; and.
The motivation behind the modification would have been to obtain a semi-supervised image semantic segmentation method that reduces the computation load of training the machine learning model. Since both WANG and ZHANG relate to training models based on labeled and unlabeled data, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHANG shows that active learning has the potential to greatly reduce the overhead of labeling data while simultaneously increasing accuracy with substantially less labeled training samples. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHANG et al. (US 20210056417 A1), Paragraph [0022].
Regarding claim 21, WANG in view of ZHANG and further in view of ZHI explicitly teach the electronic device according to claim 19,
WANG further explicitly teaches wherein the performing weighting fusion on the at least one feature value (Fig. 1. Paragraph [0036]-WANG discloses a weighted cross entropy function applied to supervised labels is defined according to Equation 1:
PNG
media_image1.png
68
523
media_image1.png
Greyscale
where L.sub.sup represents the standard term on labeled training data, capturing how well the model predicted results agree with the ground truth labels. Let θ be the parameters of a convolutional neural network, let M.sub.θ(I) be the result produced by the model for image I, and let S be the ground truth segmentation (wherein L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) is the feature value and the weighted cross entropy function performs weighting fusion).) to obtain the evaluation value corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the feature value is weighted in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).), comprises:
calculating a weighted sum of the respective feature values according to the respective weighting coefficients to obtain the evaluation value (Fig. 1. Paragraph [0038]-WANG discloses given an image I, the overall encoding cost for the image using a segmentation produced by the machine learning model 130 as a convolutional neural network represented as M.sub.θ(I) can be defined according to Equation 2: L.sub.unsup(M.sub.θ(I),I)=L(I|M.sub.θ(I))+L(M.sub.θ(I)) Equation 2 where L.sub.unsup applies to unlabeled data within the unlabeled dataset 120, capturing how well the model predicted segmentation agrees with the image under bottom-up segmentation rules (wherein L.sub.unsup is an evaluation value corresponding to the first unlabeled data).) corresponding to the first unlabeled data (Fig. 2. Paragraph [0043-0044]-WANG discloses the combined dataset 160 is a result of the machine learning model 130 being a convolutional neural network defined according to Equation 6:
PNG
media_image2.png
64
412
media_image2.png
Greyscale
Let L(θ) represent the combined dataset 160 with θ being the parameters of the convolutional neural network. Let M.sub.θ(I) be the result produced by the machine learning model 130 for image I. Let S.sub.i represent the ground truth segmentation. Where Σ.sub.i=1.sup.n L.sub.sup(M.sub.θ(I.sub.i), S.sub.i) represents the samples which agree with a top-down segmentation grouping rules and λΣ.sub.j=i.sup.m L.sub.unsup(M.sub.θ(I.sub.j),I.sub.j) represents the samples which agree with a bottom-up segmentation grouping rule. Let λ represent a predetermined weight for the unsupervised labels. Further in paragraph [0045]-WANG discloses the bottom-up grouping rules are applied to evaluate the pseudo labeled dataset 140 by assessing how well samples produced by the machine learning model 130 agree with a selected bottom-up grouping rules within the bottom-up grouping rules 138. In some embodiments, the bottom-up evaluation results 150 includes samples applied with a minimum description length principle used as the bottom-up grouping rules. For example, the samples can be derived using Equation 2 (wherein the weighted sum is performed in equation 6 to obtain a combined data set, and wherein the combined data set with the feature value is used to obtain the evaluation value).).
WANG in view of ZHANG and further in view of ZHI fail to explicitly teach acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and
However, ZHAO explicitly teaches acquiring respective weighting coefficients corresponding to respective feature values (Fig. 4A. Paragraph [0070]-ZHAO discloses the apparatus 400 includes a loss determination unit 401 configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an updating unit 402 configured to perform an updating operation for parameters of the neural network model based on the loss data (wherein the parameters of the neural network updated based on the loss data of the extracted features are weighting coefficients corresponding to respective feature values).),
wherein the weighting coefficients (Fig. 4A. Paragraph [0070]-ZHAO discloses the apparatus 400 includes a loss determination unit 401 configured to determine loss data for features extracted from a training image set using the neural network model and a loss function with a weight function, and an updating unit 402 configured to perform an updating operation for parameters of the neural network model based on the loss data (wherein the parameters of the neural network updated based on the loss data of the extracted features are weighting coefficients corresponding to respective feature values).) are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model (Fig. 1. Paragraph [0084]-ZHAO discloses it is possible to set two initial values for one parameter of a weight function, and then use each parameter value to perform iterative loss data determination and updating operations, and after one round of training for each parameter completes, a parameter value that leads to a better training result (for example, the loss data caused by the trained model) is chosen, and two parameter values around the chosen parameter value are set as parameters for the weight function used in the next round of training operation. Such process is repeated until the predetermined number of adjustments is reached or the training result is no longer better. Further in paragraph [0047]-ZHAO discloses the obtained classification probabilities are compared with real situation values 0, 1, 0, . . . , 0 (where 1 indicates the true value) to determine the difference therebetween, such as cross entropy, as the loss data.); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of WANG in view of ZHANG and further in view of ZHI of an electronic device, comprising a processor and a memory in communication connection with the processor, wherein the memory stores computer-executable instructions; and the processor is configured to execute the computer-executable instructions stored in the memory to implement an image semantic segmentation model optimization method, which comprises acquiring first unlabeled data, and evaluating the first unlabeled data based on a pre-trained target semantic segmentation model to obtain an evaluation value corresponding to the first unlabeled data, wherein the evaluation value represents effectiveness of training the target semantic segmentation model with the first unlabeled data; optimizing the target semantic segmentation model based on the target labeled data to obtain an optimized semantic segmentation model with the teachings of ZHAO of acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and.
Wherein having WANG’s device for semi-supervised image semantic segmentation method having acquiring respective weighting coefficients corresponding to respective feature values, wherein the weighting coefficients are each determined based on a variation amount of a cross entropy loss corresponding to the target semantic segmentation model; and.
The motivation behind the modification would have been to obtain a device that processes semi-supervised image semantic segmentation method that reduce the computation load of training the machine learning model. Since both WANG and ZHAO relate to training image processing machine learning models with loss functions, wherein WANG the image segmentation system described herein improves the efficiency and accuracy of a semantic segmentation model because it does not require additional labeled data nor does it require additional computational resources, while ZHAO there is a need for an improved technique to improve the training of object recognition models. Please see WANG et al. (US 20210166150 A1), Paragraph [0022], and ZHAO et al. (US 20210241097 A1), Paragraph [0006].
Allowable Subject Matter
Claims 3, 6, 20, and 23 along with their dependent claims, claims 7-8, 14, 16, and 18, are therefrom objected to as being dependent upon rejected base claims, claims 1 and 10, respectively but would be allowable if rewritten in independent form including all of the limitations of the base claims and any intervening claims, once the claim objections are overcome.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, the prior arts fail to explicitly teach wherein the feature value comprises at least one selected from a group consisting of an information entropy evaluation value, a difficulty evaluation value, a diversity evaluation value, and a consistency evaluation value; the information entropy evaluation value is configured to represent an amount of information in the first unlabeled data; the difficulty evaluation value is configured to represent a prediction difficulty of the target semantic segmentation model for the first unlabeled data; the diversity evaluation value is configured to represent a prediction difference between the first segmentation result image and the second segmentation result image; and the consistency evaluation value is configured to represent a divergence distance between the first segmentation result image and the second segmentation result image, as claimed in claim 3.
Regarding claim 6, the prior arts fail to explicitly teach inputting the target labeled data and the corresponding second unlabeled data to the first codec network to obtain a first labeled segmentation result image and a first unlabeled segmentation result image output by the first codec network; inputting the target labeled data and the corresponding second unlabeled data to the second codec network to obtain a second labeled segmentation result image and a second unlabeled segmentation result image output by the second codec network; obtaining an unsupervised loss based on the first unlabeled segmentation result image and the second unlabeled segmentation result image, as claimed in claim 6.
Regarding claim 20, the prior arts fail to explicitly teach wherein the feature value comprises at least one selected from a group consisting of an information entropy evaluation value, a difficulty evaluation value, a diversity evaluation value, and a consistency evaluation value; the information entropy evaluation value is configured to represent an amount of information in the first unlabeled data; the difficulty evaluation value is configured to represent a prediction difficulty of the target semantic segmentation model for the first unlabeled data; the diversity evaluation value is configured to represent a prediction difference between the first segmentation result image and the second segmentation result image; and the consistency evaluation value is configured to represent a divergence distance between the first segmentation result image and the second segmentation result image, as claimed in claim 20.
Regarding claim 23, the prior arts fail to explicitly teach inputting the target labeled data and the corresponding second unlabeled data to the first codec network to obtain a first labeled segmentation result image and a first unlabeled segmentation result image output by the first codec network; inputting the target labeled data and the corresponding second unlabeled data to the second codec network to obtain a second labeled segmentation result image and a second unlabeled segmentation result image output by the second codec network; obtaining an unsupervised loss based on the first unlabeled segmentation result image and the second unlabeled segmentation result image, as claimed in claim 23.
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
BALAZEVIC et al. (US 20230177344 A1) - Provided is a computer-implemented method for training a machine learning (ML) model using labelled and unlabelled data, the method comprising obtaining a set or training data comprising a set of labelled data items and a set of unlabelled data items, training a loss module of the ML model using labels in the set of labelled data items, to generate a trained loss module capable of estimating a likelihood of a label for a data item, and training a task module of the ML model using the loss module, the set of labelled data items, and the set of unlabelled data items, to generate a trained task module capable of making a prediction of a label for input data…Abstract, Fig. 3.
ROTH et al. (US 11816185 B1) - Volumetric quantification can be performed for various parameters of an object represented in volumetric data. Multiple views of the object can be generated, and those views provided to a set of neural networks that can generate inferences in parallel. The inferences from the different networks can be used to generate pseudo-labels for the data, for comparison purposes, which enables a co-training loss to be determined for the unlabeled data. The co-training loss can then be used to update the relevant network parameters for the overall data analysis network. If supervised data is also available then the network parameters can further be updated using the supervised loss…Abstract, Fig. 5.
HAVÍR et al. (US 11636602 B1) – One embodiment of the present invention sets forth a technique for performing a labeling task. The technique includes generating a multi-scale representation of an image as input to a machine learning model. The technique also includes performing one or more operations that apply the machine learning model to the multi-scale representation of the image to produce a semantic segmentation comprising predictions of labels for regions of pixels in the image. The technique further includes outputting, in a user interface, the semantic segmentation for use in assisting a user in specifying the labels for the pixels in the image…Abstract, Fig. 4.
WANG et al. (US 20190164290 A1) – Techniques related to implementing fully convolutional networks for semantic image segmentation are discussed. Such techniques may include combining feature maps from multiple stages of a multi-stage fully convolutional network to generate a hyper-feature corresponding to an input image, up-sampling the hyper-feature and summing it with a feature map of a previous stage to provide a final set of features, and classifying the final set of features to provide semantic image segmentation of the input image.…Abstract, Fig. 1.
DU et al. (US 20230196117 A1) – Embodiments of this application disclose a training method for a semi-supervised learning model which can be applied to computer vision in the field of artificial intelligence. The method includes: first predicting classification categories of some unlabeled samples by using a trained first semi-supervised learning model, to obtain a prediction label; and determining whether each prediction label is correct in a one-bit labeling manner, and if prediction is correct, obtaining a correct label (a positive label) of the sample, or if prediction is incorrect, excluding an incorrect label (a negative label) of the sample. Then, in a next training phase, a training set (a first training set) is reconstructed based on the information, and an initial semi-supervised learning model is retrained based on the first training set, to improve prediction accuracy of the model. In one-bit labeling, an annotator only needs to answer “yes” or “no” for the prediction label.…Abstract, Fig. 2 and 5.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights