DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the Application No. 18/591,205 filed February 29th, 2024.
Priority
Acknowledgment is made of applicant’s status as a continuation-in-part (CIP) of International Application No. PCT/CN2023/102430 filed on June 26th, 2023, which claims priority to foreign Patent Application No. CN 202210872051.8, filed on July 19th, 2022.
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed as foreign Patent Application No. CN 202210872051.8, filed on July 19th, 2022.
Information Disclosure Statement
The information disclosure statement(s) (IDS(s)) submitted on April 19th, 2024, and January 17th, 2025, is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS(s) is/are being considered and attached by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-8, 10-16, and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim(s) 2, 10, and 18, this/these claim(s) recite(s) the limitation “determining the unlabeled images, wherein the number of the unlabeled images is more than one”. There is insufficient antecedent basis for the phrase(s) “the unlabeled images” and “the number of the unlabeled images” in this/these limitations in this/these claim(s) because claims 1, 9, and 17 recite a singular “unlabeled image”. For examination purposes, this limitation in this/these claim(s) will be read as “determining [[the ]]unlabeled images, wherein [[the]]a number of the unlabeled images is more than one”. Furthermore, claims 3-8, 11-16, and 19-20 inherit this insufficient antecedent basis in view of their dependency to this/these claims.
Regarding claim(s) 3, 11, and 19, each of this/these claim(s) recites the limitations “the second classification reference information of the first unlabeled image” and “the first classification reference information of the second unlabeled image”. There is insufficient antecedent basis for this limitation in this/these claim(s). For examination purposes, this limitation in this/these claim(s) will be read as “[[the]]a second classification reference information of the first unlabeled image” and “[[the]]a first classification reference information of the second unlabeled image”, respectively. Furthermore, claims 4-7, 12-15, and 20 inherit this insufficient antecedent basis in view of their dependency to this/these claims.
Regarding claim(s) 4, 12, and 20, each of this/these claim(s) recites the limitation “the first classification reference information of the first unlabeled image”. There is insufficient antecedent basis for this limitation in this/these claim(s). For examination purposes, this limitation in this/these claim(s) will be read as “[[the]]a first classification reference information of the first unlabeled image”. Furthermore, claims 5-6 and 13-14 inherit this insufficient antecedent basis in view of their dependency to claims X, respectively.
Regarding claim(s) 7 and 15, each of this/these claim(s) recites the limitation “the first classification reference information of the first unlabeled image”. There is insufficient antecedent basis for this limitation in this/these claim(s). The examiner notes claims 4 and 12 preceding this/these claims previously recite this limitation. Therefore, for examination purposes, claims 7 and 15 will be read as being dependent on claims 4 and 12, respectively, such that claims 7 and 15 respectively recite: “The method according to claim [[3]]4…” and “The electronic device according to claim [[11]]12…”. as disclosed in para. [X] of the specification of the instant application.
Regarding claim(s) 14, this/these claim(s) recites the limitation “determine the loss weight of the first unlabeled image according to the intersection ratio and the comparison result”. There is insufficient antecedent basis for the phrases “the loss weight”, “the intersection ratio”, and “the comparison result” in this limitation in this/these claim(s). The examiner notes claim 13 preceding this/these claims previously recite these phrases in this limitation. Therefore, for examination purposes, claim 14 will be read as being dependent on claim 13, such that claim 14 recites: “The electronic device according to claim [[11]]13…”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 8, 9-12, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wasnik et al. (Wasnik; US 2023/0281980 A1).
Regarding claim 1, Wasnik discloses a method for training a model, comprising:
obtaining an image set used for training a model, the image set comprises a labeled image, an unlabeled image and a category label of the labeled image (para(s). [0013], [0019], [0034], [0045], and [0049], recite(s)
[0013] “…Each training sample may include at least one image of each object of the one or more objects to be detected. Furthermore, each training sample may include a class label associated with the object in the corresponding image and co-ordinates of a bounding box that includes the object in the corresponding image.”
[0019] “The system 102 may include suitable logic, circuitry, and interfaces that may be configured to train the student neural network 110 for an object detection task. The object detection task may be a semi-supervised machine learning task in which the number of training examples may be few in number (e.g., less than 4-5 images) for a target object class on which the student neural network 110 needs to be trained. Examples of the system 102 may include, but are not limited to, a computing device, a mainframe machine, a server, a computer workstation, a gaming device, and/or a consumer electronic (CE) device.”
[0034] “Each image of the set of labelled images 118A may be labeled (or annotated) with a name of an object included in the corresponding image. For example, if the image is of a dog, then the image may be labelled as a dog. In an embodiment, the image may be further labelled with co-ordinates of a bounding box that includes the object. Each unlabeled image 118B may not include any labels for the object(s) included in the image.”
[0045] “FIG. 3 is a diagram that illustrates an exemplary architecture of a teacher student framework for an end-to-end semi-supervised object detection, in accordance with an embodiment of the disclosure. FIG. 3 is explained in conjunction with elements from FIG. 1 and FIG. 2 . With reference to FIG. 3 , there is shown a diagram 300 of a teacher-student framework 302. The teacher-student framework 302 may include a teacher neural network 304 and a student neural network 306. The teacher neural network 304 may be a pretrained network for an object detection task and the student neural network 306 may be an untrained network that may have to be trained for the object detection task. With reference to FIG. 3 , there is further shown a labeled image 308 and an unlabeled image 310.”
[0049] “The input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. …”
, where the “teacher-student framework” is a model trained on an image set comprising at least a labeled image (e.g., “labeled image”), an unlabeled image (e.g., “first” and/or “second” “unlabeled image”), and a category label of the labeled image (e.g., “class label” of a “target object class”));
discriminating the labeled image and the unlabeled image by a target sub-model of the model, and determining first classification reference information of the labeled image and first classification reference information of the unlabeled image (para(s). [0050], recite(s):
[0050] “Upon generation of the input batch 312, the system 102 may be configured to apply the teacher neural network 304 on each image of the input batch 312. As discussed, the teacher neural network 304 may be pretrained network for an object detection task. Based on the application of the teacher neural network 304 on each image of the input batch 312, the system 102 may generate a first result for each input. The first result for the object (i.e., players) in the first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object (and/or other foreground or background objects) and a set of scores corresponding to the set of candidate bounding boxes. Similarly, the first result for the object (i.e., players) in the second unlabeled image 312B of the input batch 312 may include a first set of candidate bounding boxes for the object (and/or other foreground or background objects) and a first set of scores corresponding to the first set of candidate bounding boxes. Also, the first result for the object (i.e., the animal 308A) in the third labelled image 312C of the input batch 312 may include a second set of candidate bounding boxes for the object (and/or other foreground or background objects) and a second set of scores corresponding to the second set of candidate bounding boxes.”
, where generating distinct results for the “unlabeled image 312A” and labeled image 312C” is discriminating the labeled image and the labeled image by a target sub-model (e.g., a “teacher neural network”) of the model; and determining a “set of scores corresponding to the set of candidate bounding boxes” for each of the “labeled image” and the “unlabeled image[s]” is determining first classification reference information of the labeled image and the first classification information of the unlabeled image, respectively);
discriminating the unlabeled image by a non-target sub-model of the model, and determining second classification reference information of the unlabeled image (para(s). [0039] and [0073], recite(s)
[0039] “…By the application of the student neural network 110 on the first unlabeled image, the circuitry 104 may generate a second result. The second result may include a bounding box prediction for the object. As discussed, the student neural network 110 may be an untrained network that may have to be trained for the object detection task. Based on the selected foreground bounding box and the bounding box prediction, the circuitry 104 may compute a training loss over the input batch 120. The computed training loss may include a loss component for each image of the input batch 120. Based on the training loss, the circuitry 104 may train the student neural network 110 on the object detection task. Details about the training of the student neural network 110 are provided, for example, in FIG. 3.”
[0073] “In order to train the student neural network 306, image(s) from the input batch 312 may be fed to the student neural network one at a time and respective loss may be computed. In accordance with an embodiment, the system 102 may be configured to generate a second result. The generated second result may include a bounding box prediction for the object and may be generated by an application of the student neural network 306 on the first unlabeled image 312A. As discussed earlier, the student neural network 306 may be an untrained network that may have to be trained for the object detection task. Similar to the generation of the second result, the system 102 may also generate a third result by an application of the student neural network 306 on the labeled image 312C of the input batch 312.”
, where generating a result for at least a “first unlabeled image” is discriminating the unlabeled image by a non-target sub-model (e.g., a “student neural network”) of the model and determining a second classification reference information (e.g., “bounding box prediction for the object”) of the unlabeled image (e.g., a “first unlabeled labeled image”));
determining a classification loss of the target sub-model based on the first classification reference information of the labeled image, the category label of the labeled image(para(s). [0137] and [0074], recite(s)
[0137] “After computing individual losses, the system 102 may compute a training loss 324 over the input batch 312. In an embodiment, the training loss 324 may be computed based on the foreground bounding box and the bounding box prediction. In another embodiment, the training loss 324 may be computed based on the computation of the total supervised loss 318 for the labeled image 312C of the input batch. In another embodiment, the training loss 324 may be computed based on the computation of the first unsupervised loss and the second unsupervised loss. Mathematically, the computed training loss 324 may be represented using an equation (14)…”
[0074] “The system 102 may compute a total supervised loss 318 for the first result associated with the labeled image 312C and the third result associated with the labeled image 312C, by using a supervised loss function and a supervised regression loss function. In an embodiment, the total supervised loss 318 includes a supervised classification loss and a supervised box-regression loss. The supervised classification loss may be associated with the supervised loss function and the supervised box-regression loss may be associated with the supervised box-regression loss. …”
, where the “training loss 324” is a classification loss of the target sub-model based on at least the first classification reference information of the labeled image and the category label of the labeled image (e.g., “total supervised loss 318” includes “the labeled image 312C”, which includes a category label) and the second classification reference information of the unlabeled image (e.g., a “first unsupervised loss” includes at least “bounding box” determinations for at least a “first unlabeled image”)); and
tuning a model parameter of the model according to the classification loss (para(s). [0143], recite(s)
[0143] “The system 102 may be configured to train the student neural network 306 on the object detection task based on the computed training loss 324. Specifically, the computed training loss 324 may be used in a backpropagation operation to update weights parameters of the student neural network 306. To train (or re-train) the student neural network 306, the system 102 may update weight parameters of the student neural network 306 using the computed training loss 324.”
, where “updat[ing] weight parameters of the student network 306” is tuning model parameters of the model according to at least the classification loss (e.g., “computed training loss 324”)).
Where Wasnik does not explicitly disclose
determining a classification loss of the target sub-model based on …the second classification reference information of the unlabeled image;
Wasnik teaches in a further embodiment
determining a classification loss of the target sub-model based on …the second classification reference information of the unlabeled image (para(s). [0171-0172], recite(s)
[0171] “In accordance with an embodiment, the circuitry 104 may be further configured to compute the first unsupervised loss 320 for the first result that may be generated for the first unlabeled image 312A. The circuitry 104 may be further configured to compute the second unsupervised loss 322 for the first result that may be generated for the second unlabeled image 312B of the input batch 312. Each of the first unsupervised loss and the second unsupervised loss is computed by using an unsupervised loss function and includes an unsupervised classification loss and an unsupervised box-regression loss.”
[0172] “In accordance with an embodiment, the circuitry 104 may generate a second result that includes a bounding box prediction for the object. The second result may be generated by an application of the student neural network 306 on the first unlabeled image 312A. The student neural network 306 may be an untrained network that is to be trained for the object detection task.”
, where the classification loss (e.g., “training loss 324”) includes a “first unsupervised loss 320 for the first result that may be generated for the first unlabeled image 312A”; a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that generating a “second result” by the “student network 306 on the first unlabeled image 312A” would result in an unsupervised loss for the “second result” generated for the “first unlabeled image 312A”).
Since Wasnik discloses the classification loss includes second classification reference information of an unlabeled image through unsupervised learning (para(s). [0094] and [0100], recite(s)
[0094] “In an embodiment, the unsupervised classification loss function used in each of the first unsupervised loss 320 and the second unsupervised loss 322 may be equal to a sum of a foreground classification loss, a background classification loss, a background similarity loss, and a foreground-background dissimilarity loss. …”
[0100] “The foreground classification loss may help the teacher-student framework 302 to classify the foreground bounding boxes (i.e., bfg) generated by the application of the student neural network 306 on the input batch 312 from the foreground bounding boxes generated by the teacher neural network 304 on the input batch 312. Specifically, the foreground classification loss may be associated with the second unlabeled image 312B of the input batch 312. …”
, where the “unsupervised loss” is unsupervised learning and includes the second classification reference information of “bounding boxes” generated by the “student neural network” for at least the “second unlabeled image 312B”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Wasnik to incorporate basing the classification loss of the target sub-model further on the second classification reference information of the labeled image (i.e., the “first unlabeled image”) to improve the training of the model by further including second classification reference information (e.g., “second results”) of the non-target sub-model (e.g., “student neural network”) for other unlabeled images (e.g., “first unlabeled image 312A”) into the classification loss.
Regarding claim 2, Wasnik discloses the method according to claim 1, wherein the method further comprises:
obtaining an initial unlabeled image (para(s). [0049], recite(s)
[0049] “The input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. The first unlabeled image 312A may be associated with a second data augmentation type (i.e., a strong data augmentation). Whereas the second unlabeled image 312B and the labeled image 312C may be associated with the first data augmentation type (i.e., a weak data augmentation). Specifically, the first unlabeled image 312A may be generated based on application of at least one operation of the first subset of image transformations on the unlabeled image 310. The second unlabeled image 312B may be generated based on the application of at least one operation of the second subset of image transformations on the unlabeled image 310. …”
, where “unlabeled image 310” is an initial unlabeled image); and
augmenting the initial unlabeled image via different augmentation strategies and determining the unlabeled images, wherein the number of the unlabeled images is more than one, and each of the unlabeled images corresponding to one of the different augmentation strategies (para(s). [0049]—see preceding citation above—, where the “first unlabeled image” and “second unlabeled image” are at least more than one unlabeled images determined via augmenting the initial unlabeled image (e.g., “unlabeled image 310”); and para(s). [0036] further recite(s) the at least more than one unlabeled images correspond to one of different augmentation strategies:
[0036] “The circuitry 104 may generate an input batch (e.g., the input batch 120) by an application of a set of image transformations on the labeled and unlabeled images. In an embodiment, the set of image transformations may include a first subset of image transformations that may be associated with a first data augmentation type (i.e., a weak data augmentation) and a second subset of image transformations that may be associated with a second data augmentation type (i.e., a strong data augmentation). The second data augmentation type may be different from the first data augmentation type.”
, where “weak data augmentation” and “strong data augmentation” are different augmentation strategies).
Regarding claim 3, Wasnik discloses the method according to claim 2, wherein the unlabeled images comprise a first unlabeled image and a second unlabeled image (para(s). [0049]—see citation in the first limitation of claim 2 above—, where the “first unlabeled image” and “second unlabeled image” are a first and second unlabeled image, respectively), and an augmentation strategy of the first unlabeled image is lighter than an augmentation strategy of the second unlabeled image (para(s). [0049]—see citation in the first limitation of claim 2 above—, where a “weak data augmentation” is lighter than a “strong data augmentation”);
wherein determining the classification loss of the target sub-model based on the first classification reference information of the labeled image, the category label of the labeled image, and the second classification reference information of the unlabeled image comprises:
generating a first pseudo label of the first unlabeled image according to a second classification reference information of the first unlabeled image (para(s). [0073]—see citation in claim 1 limitation “discriminating the unlabeled image…” above—, where the selected “bounding box prediction” (e.g., a “foreground bounding box”) is a first pseudo label of the first unlabeled image (e.g., “unlabeled image 312A”) according to the second classification information (e.g., “score associated with each of the subset of candidate bounding boxes”) as disclosed in para(s). [0061-0063]:
[0061] “After the determination of the threshold score, the system 102 may apply the adaptive threshold filter. In accordance with an embodiment, the adaptive threshold filter may be applied after the application of the non-maximum suppression operation. The application of the adaptive threshold filter may include an operation to compare the score associated with each of the subset of candidate bounding boxes with the determined threshold score. In case the score is greater than the adaptive threshold, then a corresponding bounding box may be included in a first subset of candidate bounding boxes. Otherwise, the corresponding bounding box may be eliminated or discarded from and may not be included in the first subset of candidate bounding boxes. In some embodiments, the first subset of candidate bounding boxes may be referred to as pseudo bounding boxes.”
[0062] “It should be noted that the adaptive threshold filter may be introduced to help the teacher-student framework 302 to retain better pseudo bounding boxes. Such pseudo bounding boxes may be further used with the classification loss function.”
[0063] “The system 102 may be further configured to select a foreground bounding box. The foreground bounding box may be selected from the extracted first subset of candidate bounding boxes. In an embodiment, the selected bounding box may have a maximum score among all the candidate bounding boxes of the first subset of candidate bounding boxes. Specifically, the foreground bounding box may include a maximum portion or whole object among all other bounding boxes of the first subset of candidate bounding boxes. The foreground bounding box may be used as a ground truth for the set of foreground bounding boxes generated by the student neural network 110. Such a foreground bounding box may be used in calculation of the training loss on which the student neural network 110 has to be trained.”
);
determining a first unsupervised loss of the target sub-model according to a first classification reference information of the second unlabeled image and the first pseudo label (para(s). [0170-0171]—see citation in the teaching of Wasnik in claim 1 above—, where para(s). [0094], [0100], [0102], and [0104] further recite(s):
[0094] “In an embodiment, the unsupervised classification loss function used in each of the first unsupervised loss 320 and the second unsupervised loss 322 may be equal to a sum of a foreground classification loss, a background classification loss, a background similarity loss, and a foreground-background dissimilarity loss. …”
[0100] “The foreground classification loss may help the teacher-student framework 302 to classify the foreground bounding boxes (i.e., bfg) generated by the application of the student neural network 306 on the input batch 312 from the foreground bounding boxes generated by the teacher neural network 304 on the input batch 312. Specifically, the foreground classification loss may be associated with the second unlabeled image 312B of the input batch 312. As an example, the foreground classification loss may be mathematically represented using an equation (8)…”
[0102] “
N
b
f
g
represents the number of the foreground bounding boxes generated by the student neural network 306”
[0104] “
b
i
f
g
represents the ith foreground bounding box”
, where the “second unsupervised loss” includes at least a first supervised loss (e.g., “bounding box-regression loss”) of the target sub-model according to at least a first classification reference information (e.g., “first result” of the “teacher neural network”) of the second unlabeled image (e.g., “second unlabeled image 312B”) and the first pseudo label (e.g., “foreground bonding boxes generated by the student neural network” for at least the “first unlabeled image 312A”));
determining a supervised loss of the target sub-model according to the first classification reference information of the labeled image and the category label of the labeled image (para(s). [0074], recite(s)
[0074] “The system 102 may compute a total supervised loss 318 for the first result associated with the labeled image 312C and the third result associated with the labeled image 312C, by using a supervised loss function and a supervised regression loss function. In an embodiment, the total supervised loss 318 includes a supervised classification loss and a supervised box-regression loss. The supervised classification loss may be associated with the supervised loss function and the supervised box-regression loss may be associated with the supervised box-regression loss.”
, where the “total supervised loss” is a supervised loss according to at least the first classification reference information of the labeled image (e.g., “first result of the labeled image”) and the category label of the labeled image (e.g., the “labeled image” comprises a category label as recited in para. [0034]—see citation in the first limitation in claim 1 above)); and
determining the classification loss of the target sub-model according to the first unsupervised loss of the target sub-model and the supervised loss of the target sub-model (para(s). [0137]—see citation in claim 1 limitation “determining a classification loss…" above—, where the “training loss” is a classification loss according to at least the first unsupervised loss (i.e., the “first unsupervised and the second unsupervised loss” comprises at least the first unsupervised loss) and the supervised loss (e.g., the “total supervised loss”) of the target sub-model).
Regarding claim 4, Wasnik discloses the method according to claim 3, wherein before determining the first unsupervised loss of the target sub-model according to the first classification reference information of the second unlabeled image and the first pseudo label, the method further comprises:
determining a loss weight of the first unlabeled image according to the first pseudo label and a second pseudo label, wherein the second pseudo label is generated according to a first classification reference information of the first unlabeled image (para(s). [0037], recite(s)
[0037] “After the input batch 120 is generated, images from input batch 120 are fed to the teacher neural network 108 and the student neural network 110. For each image of the input batch 120, the circuitry 104 may generate a first result (i.e., a supervised or an unsupervised object detection result). The first result may be generated by an application of the teacher neural network 108 on images of the input batch 120. As discussed, the teacher neural network 108 may be a pretrained network for the object detection task. For an object in a first unlabeled image of the input batch 120, the first result may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Each score of the set of scores may correspond to a confidence score associated with the corresponding candidate bounding box. Specifically, the confidence score may indicate a likelihood of presence of the object inside a corresponding bounding box.”
, where each “candidate bounding box” in the “set of candidate bounding boxes” for “an object in a first unlabeled image” are pseudo labels (e.g., one candidate bounding box is a “first pseudo label” and a second candidate bounding box is a “second pseudo label”) each generated according to at least first classification reference information of the first unlabeled image (e.g., “first result” of the “teacher neural network” from a “first unlabeled image” input); wherein each “confidence score associated with the corresponding bounding box” is a loss weight according to the corresponding pseudo label);
wherein determining the supervised loss of the target sub-model according to the first classification reference information of the labeled image and the category label of the labeled image comprises:
determining a second unsupervised loss of the first unlabeled image according to the first classification reference information of the second unlabeled image and the first pseudo label (para(s). [0081], [0088], and [0131], recite(s)
[0081] “n an embodiment, the system 102 may compute a first unsupervised loss 320 for the first result (i.e., generated for the first unlabeled image 312A of the input batch 312). Also, the system 102 may compute a second unsupervised loss 322 for the first result (i.e., generated for the second unlabeled image 312B of the input batch 312). Each of the first unsupervised loss 320 and the second unsupervised loss 322 may be computed by using an unsupervised loss function. Also, each of the first unsupervised loss 320 and the second unsupervised loss 322 may include an unsupervised classification loss and an unsupervised box-regression loss. In case of the first unsupervised loss 320, the unsupervised box-regression loss may be generated after the application of the second label generator 316 on the first result (generated for the first unlabeled image 312A of the input batch 312). …”
[0088] “In case of the second unsupervised loss 322, the unsupervised box-regression loss may be generated after the application of the second label generator 316 on the first result (generated for the second unlabeled image 312B of the input batch 312). …”
[0131] “As discussed earlier, the first unsupervised loss and the second unsupervised loss include the unsupervised box-regression loss. The unsupervised box-regression loss may provide an error between a predicted and a pseudo bounding box. …”
, where the “first unsupervised loss” includes at least a second unsupervised loss (e.g., “unsupervised box-regression loss”) of the first unlabeled image (e.g., “first unlabeled image 312A”) and the first pseudo label (i.e., a “pseudo bounding box”)); and
determining the first unsupervised loss of the target sub-model according to the loss weight and the second unsupervised loss (para(s). [0094], [0113], and [0124], recite(s)
[0094] “In an embodiment, the unsupervised classification loss function used in each of the first unsupervised loss 320 and the second unsupervised loss 322 may be equal to a sum of a foreground classification loss, a background classification loss, a background similarity loss, and a foreground-background dissimilarity loss. …”
[0113] “In an embodiment, the circuitry 104 may be configured to calculate the reliability weighting factor. The reliability weighting factor may be based on a reliability score that may be associated with jth background bounding box being the background bounding box. …”
[0124] “The foreground-background dissimilarity loss may be used to separate out the foreground and background bounding boxes generated using the student neural network 306. In an embodiment, the foreground-background dissimilarity loss may follow a principle of relativistic average discriminator loss function, which is used to match two different probability distribution. The foreground-background dissimilarity loss may provide a dissimilarity between background scored and foreground scores associated with the background bounding boxes and foreground bounding boxes (generated by the student neural network 306). …”
, where determining the “second unsupervised loss” includes determining the “reliability weight factor… associated with ith background bounding box” and “background scored and foreground scores” of “foreground and background bounding boxes” is determining the first unsupervised loss of the target sub-model according to the loss weight (e.g., “weight factor[s]”) and the second unsupervised loss (e.g., “bounding box-regression loss” of either “foreground” or “background” bounding boxes)).
Regarding claim 8, Wasnik discloses the method according to claim 1, wherein the target sub-model comprises a first sub-model and a second sub-model (para(s). [0050] and [0039]—see citations in claim 1 limitation “discriminating the labeled image and unlabeled image by a target sub-model…” and “discriminating the unlabeled image by a non-target sub-model…”—, wherein the “teacher neural network” is also a first sub-model and the “student neural network” is also a second sub-model),
wherein tuning the model parameter of the model according to the classification loss comprises:
summing a first classification loss of the first sub-model and a second classification loss of the second sub-model by using a weighted summation and determining the classification loss of the model (para(s). [0137] and [0074]—see citations in claim 1 limitation “determining a classification loss…” above—, where the “total supervised loss… represented using an equation (14)” is a weighted summation of a first classification loss of the first sub-model (e.g., “first unsupervised loss”) and a second classification loss of the second sub-model (e.g., “second unsupervised loss”) as depicted in equation 14; wherein the “first unsupervised loss” and “second unsupervised loss” are classification losses as each unsupervised loss comprises at least an “unsupervised classification loss” as recited in para(s). [0081]:
[0081] “In an embodiment, the system 102 may compute a first unsupervised loss 320 for the first result (i.e., generated for the first unlabeled image 312A of the input batch 312). Also, the system 102 may compute a second unsupervised loss 322 for the first result (i.e., generated for the second unlabeled image 312B of the input batch 312). Each of the first unsupervised loss 320 and the second unsupervised loss 322 may be computed by using an unsupervised loss function. Also, each of the first unsupervised loss 320 and the second unsupervised loss 322 may include an unsupervised classification loss and an unsupervised box-regression loss. …”
); and
tuning the model parameter of the model according to the first classification loss and the second classification loss by using a back propagation algorithm (para(s). [0143]—see citation in claim 1 limitation “tuning a model parameter…” above—, where “updat[ing] weight parameters of the student network 306” is tuning model parameters of the model).
Regarding claim 9, the claim differs from claim 1 in that the claim is in the form of an electronic device comprising: a storage device; at least one processor; and the storage device storing one or more programs, which when executed by the at least one processor, cause the at least one processor to perform the method of claim 1. Wasnik discloses said storage device and processor (para(s). [0020-0021] and , recite(s)
[0020] “The circuitry 104 may include suitable logic, circuitry, and interfaces that may be configured to execute program instructions associated with different operations to be executed by the system 102. The circuitry 104 may be implemented based on a number of processor technologies known in the art. Examples of the processor technologies may include, but are not limited to, a Central Processing Unit (CPU), an x86-based processor, a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphical Processing Unit (GPU), a co-processor (such as an inference accelerator or an Artificial Intelligence (AI) accelerator), and/or a combination thereof.”
[0021] “The memory 106 may include suitable logic, circuitry, and/or interfaces that may be configured to store the program instructions executable by the circuitry 104. …Examples of implementation of the memory 106 may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or a Secure Digital (SD) card.”
). Therefore, claim 9 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 10, the claim recites similar limitations to claim 2 and is rejected for similar rationale and reasoning (see the analysis for claim 2 above).
Regarding claim 11, the claim recites similar limitations to claim 3 and is rejected for similar rationale and reasoning (see the analysis for claim 3 above).
Regarding claim 12, the claim recites similar limitations to claim 4 and is rejected for similar rationale and reasoning (see the analysis for claim 4 above).
Regarding claim 16, the claim recites similar limitations to claim 8 and is rejected for similar rationale and reasoning (see the analysis for claim 8 above).
Regarding claim 17, the claim differs from claim 1 in that the claim is in the form of a non-transitory storage medium. Wasnik discloses said non-transitory storage medium (para(s). [0021]—see citation in claim 9 above). Therefore, claim 17 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 18, the claim recites similar limitations to claim 2 and is rejected for similar rationale and reasoning (see the analysis for claim 2 above).
Regarding claim 19, the claim recites similar limitations to claim 3 and is rejected for similar rationale and reasoning (see the analysis for claim 3 above).
Regarding claim 20, the claim recites similar limitations to claim 4 and is rejected for similar rationale and reasoning (see the analysis for claim 4 above).
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wasnik as applied to claims 1 and 9 above, and further in view of Chen et al. (Chen; WO 2023/088174 A1).
Regarding claim 7, Wasnik discloses the method according to claim 3, wherein the first classification reference information of the first unlabeled image and the second classification reference information of the first unlabeled image both comprise a probability that the first unlabeled image is identified as belonging to(para(s). [0037]—see citation in the first limitation in claim 4—, where the “confidence score associated with the corresponding candidate bounding box” for candidate bounding boxes output from feeding an input image (e.g., “first unlabeled image 312A”) into the “teacher neural network” and the “student neural network” are a first classification reference information and second classification reference information of the first unlabeled image, respectively; and the “confidence score” is a probability (i.e., “likelihood”); and the “confidence score” indicates that the first unlabeled image is identified as belonging to preset categories (e.g., “objects of a certain class” or “target object class”) as recited in para(s). [0011], [0013], and [0019]:
[0011] “The following described implementations may be found in a disclosed system and method for teaching student network for end-to-end semi-supervised object detection. Object detection may be defined as a task of detecting instances of objects of a certain class within an image or a video. In some cases, object detection further includes another task of generation of a bounding box around the detected object ….”
[0013] “Recently, the task of object detection is accomplished by using a neural network model (or multiple neural network models) that are pre-trained for the task of detection of a one or more objects. To train the neural network model (or multiple neural network models), a dataset must be generated that includes a plurality of training samples. Each training sample may include at least one image of each object of the one or more objects to be detected. Furthermore, each training sample may include a class label associated with the object in the corresponding image and co-ordinates of a bounding box that includes the object in the corresponding image.”
[0019] “The system 102 may include suitable logic, circuitry, and interfaces that may be configured to train the student neural network 110 for an object detection task. The object detection task may be a semi-supervised machine learning task in which the number of training examples may be few in number (e.g., less than 4-5 images) for a target object class on which the student neural network 110 needs to be trained. Examples of the system 102 may include, but are not limited to, a computing device, a mainframe machine, a server, a computer workstation, a gaming device, and/or a consumer electronic (CE) device.”
);
wherein generating the first pseudo label of the first unlabeled image according to the second classification reference information of the first unlabeled image comprises:
determining a preset category corresponding to a(para(s). [0033-0034], [0037], [0061], and [0073], recite(s)
[0033] “The image dataset 118 may correspond to a collection of instances of one or more objects and may include a set of labelled images 118A and a set of unlabeled images 118B. Each image in the image dataset 118 may include at least one object. The object may be an animate object or an inanimate object. The animate objects may possess a quality or features of a living being whereas in-animate objects may lack such features. Examples of animated objects may include humans, birds, animals, and the like. Examples of in-animated objects may include rocks, chairs, vehicles, and the like.”
[0034] “Each image of the set of labelled images 118A may be labeled (or annotated) with a name of an object included in the corresponding image. For example, if the image is of a dog, then the image may be labelled as a dog. In an embodiment, the image may be further labelled with co-ordinates of a bounding box that includes the object. Each unlabeled image 118B may not include any labels for the object(s) included in the image.”
[0037] “… For an object in a first unlabeled image of the input batch 120, the first result may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Each score of the set of scores may correspond to a confidence score associated with the corresponding candidate bounding box. Specifically, the confidence score may indicate a likelihood of presence of the object inside a corresponding bounding box.”
[0061] “After the determination of the threshold score, the system 102 may apply the adaptive threshold filter. In accordance with an embodiment, the adaptive threshold filter may be applied after the application of the non-maximum suppression operation. The application of the adaptive threshold filter may include an operation to compare the score associated with each of the subset of candidate bounding boxes with the determined threshold score. In case the score is greater than the adaptive threshold, then a corresponding bounding box may be included in a first subset of candidate bounding boxes. Otherwise, the corresponding bounding box may be eliminated or discarded from and may not be included in the first subset of candidate bounding boxes. In some embodiments, the first subset of candidate bounding boxes may be referred to as pseudo bounding boxes.”
[0073] “In order to train the student neural network 306, image(s) from the input batch 312 may be fed to the student neural network one at a time and respective loss may be computed. In accordance with an embodiment, the system 102 may be configured to generate a second result. The generated second result may include a bounding box prediction for the object and may be generated by an application of the student neural network 306 on the first unlabeled image 312A.”
, where the “bounding box prediction for the object” output by the “student neural network” from input of the “first unlabeled image 312A” includes determining a preset category (e.g., a “candidate bounding box” of a target object) corresponding to a probability (e.g., “confidence score… indicat[ing] a likelihood of presence of the object inside a corresponding bounding box”) from the preset categories (e.g., the class label of the target object) according to the second classification reference information (e.g., “second result” or “bounding box prediction”) of the first unlabeled image); and
generating the first pseudo label of the first unlabeled image according to the preset category preset probability threshold (para(s). [0061] and [0073]—see citation in the preceding limitation immediately above—, where determining a “pseudo bounding box” based on the confidence score of the “candidate bounding box” is “greater than the adaptive threshold” is generating a first pseudo label of at least the first unlabeled image (e.g., “first unlabeled image 312A”) according to the preset category (i.e., the confidence “score” is based on the target “class label” corresponding to the predicted “bounding box”) in response that the probability (e.g., “confidence score”) is greater than a preset probability threshold (e.g., “adaptive threshold”)).
Where Wasnik does not specifically disclose
wherein the first classification reference information… and the second classification reference information… both comprise a probability that the first unlabeled image is identified as belonging to each of a plurality of the preset categories;
determining a preset category corresponding to a maximum probability from the plurality of preset categories…; and
generating the first pseudo label of the first unlabeled image according to the preset category corresponding to the maximum probability…;
Chen teaches in the same field of endeavor a first and second sub-model training using pseudo-labels
wherein the first classification reference information… and the second classification reference information… both comprise a probability that the first unlabeled image is identified as belonging to each of a plurality of the preset categories (description, para(s). [0039], [0050], [0063-0064], [0067], [0073], [0092] and [0098], recite(s)
[0039] “For example, the initial learning model and the initial management model support the same number of categories to be detected. …”
[0050] “For example, a training dataset can be pre-constructed, which may include multiple labeled data (such as labeled images). For each labeled data, there is corresponding labeling information, which includes, but is not limited to, a bounding box (such as the coordinates of the four vertices of a rectangular bounding box) and a labeling category (i.e., the category of the target object within the bounding box).”
[0063] “For example, an initial management model (also known as an initial teacher model) can be generated based on the baseline model. …”
[0064] “For example, an initial learning model (also called an initial student model) can be generated based on the baseline model. …”
[0067] “For example, for each unlabeled data in the sample dataset, the unlabeled data can be input into the initial management model, which processes the unlabeled data to obtain the prediction box corresponding to the unlabeled data, the prediction label corresponding to the prediction box, and the probability value corresponding to the prediction label (i.e., the probability value that the target object in the prediction box is the prediction label).”
[0073] “Assuming the initial management model supports object detection for categories 1, 2, and 3, for each pseudo-label, if the predicted label in the pseudo-label is category 1, then the pseudo-label corresponds to category 1; if the predicted label in the pseudo-label is category 2, then the pseudo-label corresponds to category 2; and if the predicted label in the pseudo-label is category 3, then the pseudo-label corresponds to category 3.
In summary, for each category that the initial management model supports for detection, all pseudo-labels (i.e., predicted bounding boxes and predicted labels) corresponding to that category can be obtained. …”
[0092] “Step 205: Perform first data augmentation on the unlabeled data, input the first data augmented unlabeled data into the initial learning model, and obtain the first predicted value corresponding to the unlabeled data; determine the first predicted label and the first predicted box based on the first predicted value corresponding to the high-quality pseudo-label, and determine the second predicted label and the second predicted box based on the first predicted value corresponding to the uncertain pseudo-label.”
[0098] “…For example, assuming the initial learning model supports detecting three categories, namely category 1, category 2, and category 3, the second predicted label can include the first probability value corresponding to category 1 (e.g., 0.5), the first probability value corresponding to category 2 (e.g., 0.3), and the first probability value corresponding to category 3 (e.g., 0.2), that is, the second predicted label is [0.5, 0.3, 0.2].”
, where determining “probability value[s]” for each category supported by either the “initial teacher model” or an “initial student model” is determining first and second classification reference information both comprising a probability (e.g., “probability value[s]”) that the first unlabeled image (e.g., “unlabeled data”) is identified as belonging to each of a plurality of preset categories (e.g., the “number of categories” supported by the each of the sub-models) );
determining a preset category corresponding to a maximum probability from the plurality of preset categories… (description, para(s). [0068], recite(s)
[0068] “For example, if the initial management model supports object detection for categories 1, 2, and 3, then after processing the unlabeled data, the initial management model can obtain the predicted bounding box and the probability vector corresponding to the predicted bounding box. For example, the probability vector can be [0.9, 0.06, 0.04]. Based on this probability vector, it can be known that the predicted label corresponding to the predicted bounding box is category 1, and the probability value corresponding to the predicted label is 0.9.”
, where the example of selecting the preset category of a pseudo-label as the preset category having the highest probability (i.e., selecting “category 1” with probability value “0.9” amongst probability values of 0.9 for category 1, 0.06 for category 2, and 0.04 for category 3) is determining a preset category corresponding to a maximum probability from the plurality of preset categories); and
generating the first pseudo label of the first unlabeled image according to the preset category corresponding to the maximum probability (description, para(s). [0072] and [0077], recite(s)
[0072] “For example, for each category that the initial management model supports detection, all pseudo-labels corresponding to that category are sorted based on the probability values of all pseudo-labels corresponding to that category (the category corresponding to the pseudo-label is known based on the predicted label of the pseudo-label). Based on the sorting results, the K pseudo-labels with the largest probability values are selected as high-quality pseudo-labels corresponding to that category, and the remaining pseudo-labels among all pseudo-labels corresponding to that category other than the high-quality pseudo-labels corresponding to that category are selected as uncertain pseudo-labels corresponding to that category.”
[0077] “The value of K can be configured based on experience, or it can be determined based on the total number of pseudo-labels corresponding to the category. For example, K equals the total number * M, where M is a value between 0 and 1, and can be configured based on experience, such as 20%, 30%, etc.
With M equal to 20%, since category 1 corresponds to 100 pseudo-labels, the value of K is 20. That is, the top 20 pseudo-labels are selected from all the pseudo-labels corresponding to category 1 as high-quality pseudo-labels, and the remaining 80 pseudo-labels are used as uncertain pseudo-labels.
Since category 2 corresponds to 200 pseudo-labels, the value of K is 40. That is, the top 40 pseudo-labels are selected from all the pseudo-labels corresponding to category 2 as high-quality pseudo-labels, and the remaining 160 pseudo-labels are used as uncertain pseudo-labels, and so on.”
, where determining “high-quality pseudo-labels corresponding to that category” by selecting “pseudo-labels with the largest probability” is generating at least a first pseudo label according to the preset category corresponding to the maximum probability (e.g., “largest probability values”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Wasnik to incorporate a probability that the first unlabeled image is identified as belonging to each of a plurality of preset categories in the first and second classification reference information of the unlabeled image, determining a preset category corresponding to a maximum probability from the plurality of preset categories according to the second classification reference information of the first unlabeled image, and generating the first pseudo label of the first unlabeled image according to the preset category corresponding to the maximum probability in response that the maximum probability is greater than a preset threshold to classify objects of different categories within a single unlabeled and selecting high-quality pseudo labels to improve the quality of pseudo labels generated for unlabeled images as taught by Chen (e.g., see para. [0092] above).
Regarding claim 15, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Allowable Subject Matter
Claims 5-6 and 13-14 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666