DETAILED ACTION
This office action is responsive to the above identified application filed 08/06/2024. The application contains claims 1-23, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Disclosure Statement with references submitted 7/1/2026, has/have been considered and entered into the file.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 limitation “training model configured to …” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Dependent claims inherit independent claim deficiency.
Claim limitations “automatic model training module” in claims 8 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. While the specification describes various modules in terms of software code, the specification is devoid for any explicit definition of the phrase “communications module”. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Claim limitations “model management module” in claims 10 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. While the specification describes various modules in terms of software code, the specification is devoid for any explicit definition of the phrase “communications module”. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Claim limitations “optimization module.” in claims 12 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. While the specification describes various modules in terms of software code, the specification is devoid for any explicit definition of the phrase “communications module”. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Claim limitation “automatic model training module for” in claims 14 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. While the specification describes various modules in terms of software code, the specification is devoid for any explicit definition of the phrase “communications module”. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph. Dependent claims inherit independent claim deficiency.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 1-23 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “rare targets” in claims 1-23 is a relative term which renders the claim indefinite. The term in claims “1-23” is not defined by the claims, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The specification disclose examples of data that could be considered as a rare target as medical or military imaging. However, this is not a limiting or clear definition that can be considered as a standard for determining what is rare or popular target. One is considered rare for one person could be considered popular for another
Regarding claim 7, the phrase "such as" renders the claim indefinite because it is unclear whether the limitations following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-13 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter. During examination, the claims must be interpreted as broadly as their terms reasonably allow. In re American Academy of Science Tech Center, 367 F.3d 1359, 1369, 70 U.S.P.Q.2d 1827, 1834 (Fed. Cir. 2004). Independent claim 1 recites a “system,” which is not comprehensively defined by the specification. The broadest reasonable interpretation of a claim drawn to a system covers software per se in view of the ordinary and customary meaning of system, particularly when the specification is silent. Software per se is not a “process,” a “machine,” a “manufacture,” or a “composition of matter” as defined in 35 U.S.C. § 101. Examiner suggests adding a recitation of a “processor.”
Double Patenting
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Claim 1-23 provisionally rejected under 35 U.S.C. 101 as claiming the same invention as that of claim 1-23 of copending Application No. 18795326 (reference application). This is a provisional statutory double patenting rejection since the claims directed to the same invention have not in fact been patented.
18795326
19727034
1. A customized self-training machine learning system for video analytics for rare targets, comprising: a data processing module; a data annotation module with a labelling module; an automatic model training module configured to self-train a machine learning model based on the user’s desire to detect rare targets; a model verification module; wherein the model verification module comprises an automatic error analysis and a label approval to optimize the machine learning model; a model deployment module coupled with a user operation interface module; and a video analysis module based on the trained rare targets.
1. A customized self-training machine learning system for video analytics for rare targets, comprising: a data processing module; a data annotation module with a labelling module; an automatic model training module configured to self-train a machine learning model based on the user’s desire to detect rare targets; a model verification module; wherein the model verification module comprises an automatic error analysis and a label approval to optimize the machine learning model; a model deployment module coupled with a user operation interface module; and a video analysis module based on the trained rare targets.
2. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the data processing module comprises an automatic data cleaning module, a data verification module, and a data enhancement module.
2. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the data processing module comprises an automatic data cleaning module, a data verification module, and a data enhancement module.
3. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a unimodal, a cross-modal, and a multimodal.
3. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a unimodal, a cross-modal, and a multimodal.
4. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises at least one built-in algorithm.
4. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises at least one built-in algorithm.
5. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises meta-learning for detecting rare targets.
5. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises meta-learning for detecting rare targets.
6. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a visualization module, an auto-tuning of hyperparameters module, a distributed training module and an automatic start or stop module.
6. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a visualization module, an auto-tuning of hyperparameters module, a distributed training module and an automatic start or stop module.
7. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the rare targets comprise scarce, non-general and confidential objects such as medical or military imaging.
7. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the rare targets comprise scarce, non-general and confidential objects such as medical or military imaging.
8. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module is configured to detect common targets and customized targets.
8. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module is configured to detect common targets and customized targets.
9. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the model deployment module comprises a data management module, a model packaging, a model management module, a labelling task management module, and a training task management module.
9. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the model deployment module comprises a data management module, a model packaging, a model management module, a labelling task management module, and a training task management module.
10. The customized self-training machine learning system for video analytics for rare targets, according to claim 9, wherein the model management module is configured to provide model recommendations for users viewing specific information based on the generated model information.
10. The customized self-training machine learning system for video analytics for rare targets, according to claim 9, wherein the model management module is configured to provide model recommendations for users viewing specific information based on the generated model information.
11. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the video analysis module comprises a model management module, an analyzing module, an acceleration module, and an optimization module.
11. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the video analysis module comprises a model management module, an analyzing module, an acceleration module, and an optimization module.
12. The customized self-training machine learning system for video analytics for rare targets, according to claim 11, wherein the optimization module is configured to execute automatic error analysis and model automatic optimization of the model to optimize the video analysis performance.
12. The customized self-training machine learning system for video analytics for rare targets, according to claim 11, wherein the optimization module is configured to execute automatic error analysis and model automatic optimization of the model to optimize the video analysis performance.
13. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the system further comprises network-attached storage (NAS) or storage area network (SAN) providing secure and large-capacity data storage.
13. The customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the system further comprises network-attached storage (NAS) or storage area network (SAN) providing secure and large-capacity data storage.
14. A method of providing customized self-training machine learning models for video analytics for rare targets, comprising of: processing data; automatically self-training a machine learning model using the processed data; wherein using meta-learning in the automatic model training for detecting rare targets; and analyzing a video based on the trained rare targets.
14. A method of providing customized self-training machine learning models for video analytics for rare targets, comprising of: processing data; automatically self-training a machine learning model using the processed data; wherein using meta-learning in the automatic model training for detecting rare targets; and analyzing a video based on the trained rare targets.
15. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the processing data comprises: uploading data; verifying and cleaning the uploaded data; enhancing the data; and labelling the data.
15. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the processing data comprises: uploading data; verifying and cleaning the uploaded data; enhancing the data; and labelling the data.
16. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the step of self-training machine learning model comprises: displaying progress of current training, remaining time, and accuracy of current training; auto-tuning of hyperparameters; distributing training; and automatically starting or stopping the training.
16. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the step of self-training machine learning model comprises: displaying progress of current training, remaining time, and accuracy of current training; auto-tuning of hyperparameters; distributing training; and automatically starting or stopping the training.
17. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 16, wherein the step of self-training machine learning model further comprises: detecting common targets and customized targets.
17. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 16, wherein the step of self-training machine learning model further comprises: detecting common targets and customized targets.
18. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises: providing preset scenarios and their mature models for video analysis models; and adopting the detection of common targets when the user’s needs are not met during the self-training of the machine learning model.
18. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises: providing preset scenarios and their mature models for video analysis models; and adopting the detection of common targets when the user’s needs are not met during the self-training of the machine learning model.
19. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises: using transfer learning; and freezing a pre-determined number of network layers for detecting customized targets with a large amount of training data.
19. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises: using transfer learning; and freezing a pre-determined number of network layers for detecting customized targets with a large amount of training data.
20. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the method further comprises verifying the trained model to optimize the machine learning model after the training is completed.
20. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the method further comprises verifying the trained model to optimize the machine learning model after the training is completed.
21. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 20, wherein the method further comprises: performing data management, model management and labeling management using a user operation interface; and performing training task management using the user operation interface after verifying the trained model.
21. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 20, wherein the method further comprises: performing data management, model management and labeling management using a user operation interface; and performing training task management using the user operation interface after verifying the trained model.
22. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the analyzing of video comprises: providing model recommendations and automatic deployment; providing at least one image analysis, video analysis and batch analysis; accelerating the machine learning model; and activating optimization module to realize automatic error analysis and model automatic optimization of the machine learning model to optimize video analysis performance.
22. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the analyzing of video comprises: providing model recommendations and automatic deployment; providing at least one image analysis, video analysis and batch analysis; accelerating the machine learning model; and activating optimization module to realize automatic error analysis and model automatic optimization of the machine learning model to optimize video analysis performance.
23. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 22, wherein the method further comprises simultaneously updating output feedback to the machine learning models.
23. The method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 22, wherein the method further comprises simultaneously updating output feedback to the machine learning models.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4, 7-13 are rejected under 35 U.S.C. 102(a)(1) and 35 U.S.C. 102(a)(2) as being anticipated by Zhdanov et al. [US11048979B1, hereinafter Zhdanov].
With regard to Claim 1,
Zhdanov teach a customized self-training machine learning system for video analytics for rare targets (Abstract, “An active learning-based data labeling service enables a user to build and manage large, high accuracy datasets for use in various machine learning systems”, Col. 3, lines 60-67, “Each dataset can be a collection of homogeneous pieces of data (such as image data, video data, comma separated values (CSV) files, etc.)”, Col. 4, lines 21-26, “for a machine learning model to recognize species of birds, the label space includes labels (and associated label classes) for various species of birds. These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)”, Col. 5, lines 15-17, “such as video data, audio data, etc. may also be used in such a system”), comprising:
a data processing module (Fig. 1, 5, Col. 5, lines 41-43, “dataset service 116 can communicate with WIS 118, machine annotation service 114, and core engine 110 to manage the flow of data to be labeled”);
a data annotation module with a labelling module (Fig. 1, 6-7, Col. 5, lines 48-52, “The WIS 118 can interface with various manual annotators including crowdsource annotators, private (e.g., in-house) annotators, and/or annotating service providers (collectively annotating service 120)”, “an annotation manager 302 can create a workflow for annotating service 120 to perform“);
an automatic model training module configured to self-train a machine learning model based on the user’s desire to detect rare targets (Claim 4, “training the machine learning model using the plurality of labels to generate an updated machine learning model, wherein the updated machine learning model is used to perform auto-labeling in a next iteration of the active learning loop”, Col. 4, lines 21-26, “for a machine learning model to recognize species of birds, the label space includes labels (and associated label classes) for various species of birds. These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)“, Col. 4, lines 27-29, “the customer can specify a workflow to be used for labeling”);
a model verification module, wherein the model verification module comprises an automatic error analysis and a label approval to optimize the machine learning model (Claim 8, “determining a plurality of accuracy values for the plurality of auto-annotations for the validation dataset, performing regression on the accuracy of the plurality of auto-annotations to determine a confidence interval for each accuracy value “, Col. 6, lines 18-21, “The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required”, Col. 6, lines 62-67, “score is compared to a desired quality threshold (which may be a default value or may be provided by the customer as an input) to determine whether consolidated labels have achieved the desired quality level and hence are considered trained labels”);
a model deployment module coupled with a user operation interface module ( Col. 13, 47-50, “returning at least one of the updated machine learning model or an output dataset, the output dataset labeled using the updated machine learning model “, Col. 8, 23-25, “The resulting labeled dataset and/or the converged model can then be output to output data store 206”, Col. 3, 27-33, “Users may interact with a provider network 100 across one or more intermediate networks 106 (e.g., the internet) via one or more interface(s) 104, such as through use of application programming interface (API) calls, via a console implemented as a website or application, etc. The interface(s) 104 may be part of, or serve as a front-end”); and
a video analysis module based on the trained rare targets (Col. 6, 46-48, ”if the updated model has converged, then the remainder of the input dataset can be accurately identified”, Col. 2, lines 64-66, “inference can be performed on the dataset in each iteration, increasing the speed at which the dataset is annotated”, Col. 10, lines 5-8, “Depending on the application, the machine learning model may perform classification, segmentation, object detection, etc.”).
With regard to Claim 2,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the data processing module comprises an automatic data cleaning module (“removing bias or low-quality annotators”, Col. 6, lines 3-7, “based on each annotators' accuracy history, their annotations can be weighted. If one annotator has a 50% accurate history, their annotations may have a lower weight than another annotator with a 100% accurate history”), a data verification module (Col. 3, line 64-66, “gold standard dataset may refer to a dataset that has been verified as being accurately labeled”, Col. 7, lines 17-19, “a random subset of the unlabeled dataset is selected for validation and sent to human annotators”), and a data enhancement module (Col. 8, lines 53-56, “output dataset stored in output data store 206 may be an updated version of the input dataset, which has been updated by combining the input dataset with the generated labels as a new dataset”, Col. 5-6, lines 67-1, “removing bias or low-quality annotators”).
With regard to Claim 4,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises at least one built-in algorithm (Col. 4, lines 54-57, “active learning techniques used may be specified by the customer in a custom workflow or may be built-in as part of data labeling service“).
With regard to Claim 7,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the rare targets comprise scarce, non-general and confidential objects such as medical or military imaging (Col. 9, lines 33-35, “dataset may be encrypted prior to it being submitted to an annotation team or service, to protect the dataset while in transit”, Col. 4, lines 21-26, “for a machine learning model to recognize species of birds, the label space includes labels (and associated label classes) for various species of birds. These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)”, Col. 4, lines 6-9, “customer may also specify one or more of a gold standard dataset, a target label space, a desired quality threshold, and an annotation budget”, Col. 4, lines 19-26, “label space may be a sub-graph of the overall graph of predefined labels that can be used for annotations for specific dataset objects. For example, for a machine learning model to recognize species of birds, the label space includes labels (and associated label classes) for various species of birds. These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)”) .
The Examiner further notes that the data content (“medical or military imaging”) is non-functional descriptive material and is not functionally involved in the steps recited. Thus, this descriptive material will not distinguish the claimed invention from the prior art in terms of patentability. See In re Gulack, 703 F.2d 1381, 218 USPQ 401, 403 (Fed. Cir. 1983); In re Lowry, 32 F.3d 1579, 32 USPQ2d 1031 (Fed. Cir. 1994).
With regard to Claim 8,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module is configured to detect common targets and customized targets (Col. 3, 36-39, “Machine annotation service 114 may use an existing model that has been trained on the same or similar labelspace which is selected for the input dataset”, Col. 4, 6-9, “customer may also specify one or more of a gold standard dataset, a target label space, a desired quality threshold, and an annotation budget”, Col. 4, 24-26, “These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)”).
With regard to Claim 9,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the model deployment module comprises a data management module (Co. 9, lines 39-42, “dataset service 116 can store annotated datasets received from manually annotators and auto-annotated datasets annotated by a machine annotator”), a model packaging (Col. 6, lines 51-54, “the data labeling service 108 can output one or more of the converged model or the labeled dataset”), a model management module (Col. 6, lines 37-40, “The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset”, Col. 6, lines 44-45, “train the model maintained by the machine annotation service”), a labelling task management module (Fig. 5, 508, Col. 12, lines 23-24, “a ListLabelingJobs command 508 that may return objects representing current labeling jobs”, Col. 4, lines 41-44, “The core engine 110 is responsible for triggering/calling functions, activities and tasks on multiple different microservices as part of the overall execution of a workflow”), and a training task management module (Col. 6, lines 40-44, “a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model“, Col. 4, lines 38-39, “The core engine 110 drives the overall workflow definition, execution, monitoring, and orchestration“).
With regard to Claim 10,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 9, wherein the model management module is configured to provide model recommendations for users viewing specific information based on the generated model information (Col. 4, lines 6-8, “the customer may also specify one or more of a gold standard dataset, a target label space, a desired quality threshold, and an annotation budget“, “Machine annotation service 114 may use an existing model that has been trained on the same or similar labelspace which is selected for the input dataset”, Fig. 5, 504, Col. 12, 16-17, “ a DescribeLabelingJob command 504 which may receive a customer ID and return projects associated with that customer ID”).
With regard to Claim 11,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the video analysis module comprises a model management module, an analyzing module (Col. 12, lines 3-4, “dataset can be passed through a machine learning model 406 to identify features of the dataset”, Col. 9-10, lines 67-2, “one or more machine learning models 406 to determine a portion of the dataset to be auto-annotated and a portion of the dataset to be manually annotated”), an acceleration module (Col. 11, lines 1-4, “This incremental training and inference allows each iteration to build on the last, increasing the speed at which inference can be performed and improving the accuracy of the model with each iteration”, Col. 2, 66-67, “Embodiments increase the speed at which auto-annotation can be performed”), and an optimization module (Fig. 4, Col. 10, lines 53-57, “the active learning loop can be performed iteratively, incrementally training a more accurate machine learning model and incrementally auto-annotating a larger portion of the input dataset 400”).
With regard to Claim 12,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 11, wherein the optimization module is configured to execute automatic error analysis and model automatic optimization of the model to optimize the video analysis performance (Fig. 4, Co. 10, 18-20, “Because the features of the validation dataset are known, the accuracy of the model can be evaluated”, Col. 13, lines 50-60, “calculating a quality score associated with the plurality of labels, determining the quality score is less than a quality threshold, and requesting additional annotation of the subset of the input dataset from the annotation service”, Col. 11, lines 48-50, “Using the new labeled portions of the input dataset, the machine learning model can be further trained”, claim 11).
With regard to Claim 13,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the system further comprises network-attached storage (NAS) or storage area network (SAN) providing secure and large-capacity data storage (Col. 17, 50-52, “network interface 940 may support communication … via storage area networks (SANs) such as Fibre Channel SANs”, Col. 19, lines 20-22, “the information may reside in a storage-area network (SAN) familiar to those skilled in the art”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Zhdanov et al. [US11048979B1, hereinafter Zhdanov] in view of Trask et al. [US 20230368036 A1, hereinafter Trask].
With regard to Claim 3,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a unimodal (Col. 5, lines 1-5, “model that takes an image as input and outputs a set of bounding boxes”, Col. 3, lines 60-63, “a collection of homogeneous pieces of data (such as image data, video data, comma separated values (CSV) files, etc.)”.
Zhdanov does not explicitly teach a cross-modal, and a multimodal.
Trask teach training module comprises a unimodal, a cross-modal, and a multimodal (¶¶5-7, “wherein the decoders and physics simulators respectively reconstruct the multimodal dataset. When a unimodal dataset comprising a single modality of data related to the physical phenomenon is received a value of the physical phenomenon is predicted according to cross-modal inference learning from encoding and decoding of the multimodal dataset”, ¶19, “recognize and take into account that scientific and engineering data often multiple heterogeneous sources (multimodal) (e.g., images, 2D data, 1D data, scalar values, time-series data, etc.)”, ¶21, “Sampling from clusters allows cross-modal generative modeling”, ¶22, “The process allows cross-modal inference using an instantiation from a single data mode from which a synthetic “cross-modal” representation of all data modes can be obtained”, ¶¶25-27, ¶43, “ Cross-modal inference corresponds to training an autoencoder jointly across modalities of data in a manner that supports generative sampling of individual modalities. The illustrative embodiments achieve this goal in a variational inference setting by: encoding data into unimodal embeddings and applying a Product of Experts model to fuse data into a multimodal posterior”, ¶48, “To facilitate cross-modal inference, unimodal embeddings are trained to reproduce the multimodal embedding”).
Zhdanov and Trask are analogous art to the claimed invention because they are from a similar field of endeavor of machine learning models training. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhdanov resulting in resolutions as disclosed by Trask with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Zhdanov as described above to offers a richer, more adaptive learning experience by addressing different learning needs, improving engagement, and enhancing knowledge transfer. This is simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 5-6, and 14-23 are rejected under 35 U.S.C. 103 as being unpatentable over Zhdanov et al. [US11048979B1, hereinafter Zhdanov] in view of “Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers” Published 2020 hereinafter D1.
With regard to Claim 5,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises detecting rare targets.
Zhdanov does not explicitly teach training module comprises meta-learning.
D1 teach automatic model training module comprises meta-learning for detecting rare targets (P. 3, “Our approach is essentially that of supervising a low-cost “student” model with a high-cost “teacher” model (or knowledge distillation [32]), and this has been broadly applied in computer vision literature”, P. 8, Col. 1, “(𝑖𝑖) Post-retraining accuracy can be roughly estimated by training on a small subset of training data for a handful of epochs.”, “We then fit the accuracy-epoch points to the a non-linear curve model from [64] using a non-negative least squares solver [3]. This model is then used to extrapolate the accuracy that would be obtained by retraining with all the data for larger number of epochs”, P. 6, “The uncertainty of 𝐴𝑇 (𝑣,𝛾,𝜆,R,I) resembles the multi armed bandits (MAB) problem [71] to maximize the expected rewards given a limited number of trials for a set of options”, “Research has also been conducted on handling catastrophic forgetting [41, 72], using limited amount of training data [67, 82], and dealing with class imbalance”).
Zhdanov and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of iterative continuous training of machine learning models to keep the models accurate and up to date. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhdanov resulting in resolutions as disclosed by D1with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Zhdanov as described above to allow the models to adjust to novel problems using only a few examples, because they have learned generalizable patterns from many prior tasks, achieves high accuracy with far less training data than traditional methods, and reduce the need for large, task-specific datasets. This is simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 6,
Zhdanov teach the customized self-training machine learning system for video analytics for rare targets, according to claim 1, wherein the automatic model training module comprises a visualization module (Col. 6, lines 59-61, “logs and metrics in a monitoring dashboard, and/or an audit trail of annotations tasks as performed by annotators”, Col. 10, lines 23-26, “This results in a chart that shows you the confidence scores grouped by decile (or other subset) on the X axis and the Y axis shows you the accuracy for that decile”), and an automatic start or stop module (Fig. 5, 500, 510, Col. 12, lines 26-31, “a StopLabelingJob command 510 that may receive an identifier associated with a labeling job and stops execution of the labeling job”, “a start labeling job command that may cause the core engine to start a labeling workflow for a given project”).
Zhdanov does not explicitly teach an auto-tuning of hyperparameters module, a distributed training module.
D1 teach automatic model training module comprises a visualization module (P. 4, Fig. 3, “Note the Pareto boundary of configurations in (3b); for every non Pareto configuration, there is at least one Pareto configuration that is better than it in both accuracy and GPU cost”), an auto-tuning of hyperparameters module (P. 4, ¶3.1, “The hyperparameters for retraining, or “retraining configurations”, influence the resource demands and accuracy.”, “various hyperparameters: number of training epochs, batch sizes, number of neurons in the last layer, number of frozen layers, and fraction of training data”, P. 8, Algorithm 2: PickConfigs, P. 2, “accuracy after the retraining and the resource demands. We have designed a micro profiler that observes the accuracy of the retraining configurations on a small subset of the training data in the retraining window with just a few epochs. It uses these observations to extrapolate the accuracies when retrained on a larger dataset for many more epochs”, P. 5, 4, “as it requires actually retraining using all the configurations. We address this challenge by designing a micro-profiling based estimator (§4.3), which profiles only a few promising configurations on a tiny fraction of the training data with early termination”), a distributed training module (D1, 5, “Our implementation uses a collection of logically distributed modules for ease of scale-out to many video streams and resources”, P. 8, 5, “Each module acts as either the Ekya scheduler, micro-profiler, or a training/inference job, and is implemented by a long-running “actor” in Ray”, P. 9, Placement onto GPUs: “fractional resources can be spanned across two discrete GPUs. To avoid the consequent expensive inter-GPU communication, Ekya first quantizes the allocations to inverse powers of two (e.g., 1/2, 1/4, 1/8). This makes the jobs amenable to packing. Ekya then allocates jobs to GPUs in descending order of demands to reduce fragmentation”) and an automatic start or stop module (P.7, “Algorithm 1 is invoked at the beginning of each retraining window, as well as on the completion of every training job during the window”).
Zhdanov and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of iterative continuous training of machine learning models to keep the models accurate and up to date. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhdanov resulting in resolutions as disclosed by D1with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Zhdanov as described above to enables faster, more accurate, and more scalable model development by automatically finding optimal settings across large, complex search spaces while leveraging multi-node GPU/TPU resources. This is simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 14,
Zhdanov teach a method of providing customized self-training machine learning models for video analytics for rare targets, comprising of: processing data (claim 4, “obtaining, by a data labeling service implemented by at least one electronic device, an input dataset, the input dataset including a plurality of objects”); automatically self-training a machine learning model using the processed data (claim 4, “training the machine learning model using the plurality of labels to generate an updated machine learning model, wherein the updated machine learning model is used to perform auto-labeling in a next iteration of the active learning loop”, Col. 7, lines 57-59, “All previously human-labeled objects are then used to train a new version of the supervised model, and the loop repeats until all the data is labeled”); and analyzing a video based on the trained rare targets (Col. 6, lines 46-47, “the remainder of the input dataset can be accurately identified”, Col. 9, lines 20-22, “which includes each object of the dataset (e.g., image file, text file, video file, video frame, audio utterance, etc.).” Fig. 1, 11, Fig. 3).
Zhdanov does not explicitly teach using meta-learning in the automatic model training for detecting rare targets.
D1 teach using meta-learning in the automatic model training for detecting rare targets (P. 3, “Our approach is essentially that of supervising a low-cost “student” model with a high-cost “teacher” model (or knowledge distillation [32]), and this has been broadly applied in computer vision literature”, P. 8, “Post-retraining accuracy can be roughly estimated by training on a small subset of training data for a handful of epochs.”, “We then fit the accuracy-epoch points to the a non-linear curve model from [64] using a non-negative least squares solver [3]. This model is then used to extrapolate the accuracy that would be obtained by retraining with all the data for larger number of epochs”, P. 6, “The uncertainty of 𝐴𝑇 (𝑣,𝛾,𝜆,R,I) resembles the multi armed bandits (MAB) problem [71] to maximize the expected rewards given a limited number of trials for a set of options”, “Research has also been conducted on handling catastrophic forgetting [41, 72], using limited amount of training data [67, 82], and dealing with class imbalance”).
Zhdanov and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of iterative continuous training of machine learning models to keep the models accurate and up to date. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhdanov resulting in resolutions as disclosed by D1with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Zhdanov as described above to allow the models to adjust to novel problems using only a few examples, because they have learned generalizable patterns from many prior tasks, achieves high accuracy with far less training data than traditional methods, and reduce the need for large, task-specific datasets. This is simply combining prior art elements according to known methods to yield predictable results, use of known technique to improve similar devices (methods, or products) in the same way, and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
With regard to Claim 15,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the processing data comprises:
uploading data (Zhdanov Col. 8, lines 3-4, “the customer (via one or more electronic devices 102) can upload an input dataset to input data store 204”, D1, P. 12, 6.5, “One may upload a sub sampled video stream to the cloud, retrain the model in the cloud, and download the model to the edge server”);
verifying and cleaning the uploaded data (Zhdanov Col. 3, lines 63-65 “gold standard dataset may refer to a dataset that has been verified as being accurately labeled”, Col. 7, lines 17-19, “a random subset of the unlabeled dataset is selected for validation and sent to human annotators”, Col. 5-6, lines 67-1, “removing bias or low-quality annotators”, D1, P. 8, “Annotating training data: For both the micro-profiling as well as the retraining, Ekya acquires labels using a “golden model””, “we obtain the (validation) accuracy of each configuration at each epoch it was trained“, “Pruning out bad configurations: Finally, Ekya’s micro profiling also prunes out those configurations for micro profiling (and hence, for retraining) that have historically not been useful”);
enhancing the data (Zhdanov Col. 8, lines 53-56, “output dataset stored in output data store 206 may be an updated version of the input dataset, which has been updated by combining the input dataset with the generated labels as a new dataset”, Col. 5-6, lines 67-1, “removing bias or low-quality annotators”); and labelling the data (Zhdanov Col. 13, lines 27-30, “consolidating the plurality of annotations to generate a plurality of labels for the second subset of the input dataset”, D1, P. 4, “Each retraining uses data samples ac cumulated since the beginning of the last retraining (referred to as the “retraining window”)”). The same motivation to combine for claim 14 equally applies for current claim.
With regard to Claim 16,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the step of self-training machine learning model comprises: displaying progress of current training, remaining time, and accuracy of current training; auto-tuning of hyperparameters (D1, P. 4, ¶3.1, “The hyperparameters for retraining, or “retraining configurations”, influence the resource demands and accuracy.”, “various hyperparameters: number of training epochs, batch sizes, number of neurons in the last layer, number of frozen layers, and fraction of training data”, P. 8, Algorithm 2: PickConfigs, P. 2, “accuracy after the retraining and the resource demands. We have designed a micro profiler that observes the accuracy of the retraining configurations on a small subset of the training data in the retraining window with just a few epochs. It uses these observations to extrapolate the accuracies when retrained on a larger dataset for many more epochs”, P. 5, 4, “as it requires actually retraining using all the configurations. We address this challenge by designing a micro-profiling based estimator (§4.3), which profiles only a few promising configurations on a tiny fraction of the training data with early termination”); distributing training (D1, 5, “Our implementation uses a collection of logically distributed modules for ease of scale-out to many video streams and resources”, P. 8, 5, “Each module acts as either the Ekya scheduler, micro-profiler, or a training/inference job, and is implemented by a long-running “actor” in Ray”, P. 9, Placement onto GPUs: “fractional resources can be spanned across two discrete GPUs. To avoid the consequent expensive inter-GPU communication, Ekya first quantizes the allocations to inverse powers of two (e.g., 1/2, 1/4, 1/8). This makes the jobs amenable to packing. Ekya then allocates jobs to GPUs in descending order of demands to reduce fragmentation”); and automatically starting or stopping the training (Zhdanov Fig. 5, 500, 510, “a StopLabelingJob command 510 that may receive an identifier associated with a labeling job and stops execution of the labeling job”, “a start labeling job command that may cause the core engine to start a labeling workflow for a given project”, D1, “P.7, “Algorithm 1 is invoked at the beginning of each retraining window, as well as on the completion of every training job during the window”). The same motivation to combine for claim 14 equally applies for current claim.
With regard to Claim 17,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 16, wherein the step of self-training machine learning model further comprises: detecting common targets and customized targets (Zhdanov, Col. 5, lines 36-39, “Machine annotation service 114 may use an existing model that has been trained on the same or similar labelspace which is selected for the input dataset”, Col. 4 lines 24-26, “These labels can be either predefined (e.g., a closed label space) or incrementally defined by annotators (e.g., an open label space)”, D1, “customized analytics and models for each stream”).
The same motivation to combine for claim 16 equally applies for current claim.
With regard to Claim 18,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises:
providing preset scenarios and their mature models for video analysis models (Zhdanov, Col. 4, lines 28-30, “The workflow may be provided by the active learning-based labeling service or may be a custom workflow specified by the user”, Col. 7, lines 21-24, “If the customer brings a pre-trained model, it can be passed along to the main loop”, D1, P. 12, “we pre-train and cache a few tens of DNNs from earlier retraining windows from the Cityscapes dataset”, P. 9, “The uniform scheduler then picks two points on the Pareto frontier as the fixed retraining configurations to represent “high” (Config 1) and “low” (Config 2) resource usage”, “The same hold-out dataset is used to customize the off-the-shelf DNN inference model”); and
adopting the detection of common targets when the user’s needs are not met during the self-training of the machine learning model (Zhdanov, Col. 7, lines 22-25, “If there is no pretrained model, but there is some prelabeled data, this data is used to train a default model for the selected modality”, Col. 9, lines 9-13, “the active learning service 112 can be bypassed on a first iteration of the active learning loop and a random sample of the input dataset can be provided to the workforce interface service 118 to be labeled”, D1, P. 2-3, “We also constraint our decisions such that the inference accuracy at any point in time does not drop below a minimum value (so that the outputs continue to remain useful to the application)”, P. 7, “among all the configurations whose accuracy is ≥ 𝑎MIN, PickConfigs picks the configuration with the highest accuracy”). The same motivation to combine for claim 17 equally applies for current claim.
With regard to Claim 19,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 17, wherein the method further comprises:
using transfer learning (Zhdanov, Col. 7, lines 4-7, “They might also bring a pre-trained supervised machine learning model, as well as some data which has already been labeled in the past”, Col. 5, lines 36-39, “Machine annotation service 114 may use an existing model that has been trained on the same or similar labelspace which is selected for the input dataset”, D1, P. 13, “Continuous learning. Machine learning literature on continuous learning adapts models as new data comes in. A common approach used is transfer learning [32, 46, 66, 68]”); and
freezing a pre-determined number of network layers for detecting customized targets with a large amount of training data (D1, P. 4, 3.1, “Retraining fewer layers of the DNN (or, “freezing” more layers) consumes lesser GPUresources, as does training on fewer data samples, but they also produce a model with lower accuracy”, P. 4, “number of frozen layers, and fraction of training data”, P. 9, “number of layers to retrain, and the fraction of data between retraining windows to use for retraining”). The same motivation to combine for claim 17 equally applies for current claim.
With regard to Claim 20,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the method further comprises verifying the trained model to optimize the machine learning model after the training is completed (Zhdanov, Fig. 1, 11, Col. 7, lines 35-36, “The main loop starts by running inference with the model on the validation dataset”, Col. 6, 46-48, “if the updated model has converged, then the remainder of the input dataset can be accurately identified. If the updated model has not converged, then a new subset of the input dataset can be identified for further labeling”, D1, P.7, “Algorithm 1 is invoked at the beginning of each retraining window, as well as on the completion of every training job during the window to reallocate resources to the other training and inference jobs”). The same motivation to combine for claim 14 equally applies for current claim.
With regard to Claim 21,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 20, wherein the method further comprises:
performing data management, model management and labeling management using a user operation interface (Zhdanov, Fig. 2, Fig. 3, Col. 8, lines 34-36, “the user can send a request to view, download, transfer, etc. the labeled dataset in output data store 206 to user interface 104”, Col. 8, lines 59-63, “a customer can create a new labeling job at numeral 1 by sending job details (e.g., input dataset information, credentials, quality threshold, etc.) to user interface 104”, Col. 12, lines 37-39, “GetLabelsForDatasetObject command which receives an object ID and returns all labels associated with that object”); and
performing training task management using the user operation interface after verifying the trained model (Zhdanov, Col. 12, lines 32-37, “a GetWorkflowOutput command, which can receive a worflow ID and return the output of the workflow including a training dataset and labels (e.g., the labeled input dataset for that workflow) or a trained model”, “The CreateLabelingJob command can cause a new labeling project to be created and coordinated by the core engine”). The same motivation to combine for claim 20 equally applies for current claim.
With regard to Claim 22,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 14, wherein the analyzing of video comprises:
providing model recommendations and automatic deployment (Zhdanov, Col. 5, lines 36-39, “an existing model that has been trained on the same or similar labelspace which is selected for the input dataset”, Col. 14, lines 10-12, “the updated detection model is used to perform auto-labeling in a next iteration of the active learning loop”, Fig. 4, D1, “we pick the cached DNN whose class distribution (vector of object class frequencies) of its training data has the closest Euclidean distance with the current window’s data”, P. 9, “Ekya can improve inference accuracy by checkpointing the model during retraining and dynamically loading it as the inference model”);
providing at least one image analysis, video analysis and batch analysis (Zhdanov, Fig. 4, Col. 10, lines 65-67, “inference can be performed in batches while iteratively training the machine learning model”, Col. 3, lines 61-62, “such as image data, video data, comma separated values (CSV) files, etc.”, D1, P. 10, “inference accuracy (%of images correctly classified)”, P.1, “DNN inference on many live video streams”);
accelerating the machine learning model (Zhdanov, Col. 11, lines 1-5, “This incremental training and inference allows each iteration to build on the last, increasing the speed at which inference can be performed and improving the accuracy of the model with each iteration”, “Embodiments increase the speed at which auto-annotation can be performed”, D1, P. 3, “train and deploy specialized and compressed DNNs”); and
activating optimization module to realize automatic error analysis and model automatic optimization of the machine learning model to optimize video analysis performance (Zhdanov, Col. 6, lines 40-45, “the accuracy of the model can be evaluated”, “a separate training service (not shown) may obtain the labeled subset of the input dataset from the annotation consolidation service and may use the labeled subset of the input dataset to further train the model“, D1, P. 9, “Ekya reactively adjusts its allocations. Every few epochs, Ekya uses the current accuracy of the model being retrained to estimate its eventual accuracy when all the epochs are complete. It updates the expected accuracy in the profile of the retraining (Γ) with the new value, and then reruns Algorithm 1 for new resource allocations”). The same motivation to combine for claim 14 equally applies for current claim.
With regard to Claim 23,
Zhdanov-D1 teach the method of providing customized self-training of machine learning models for video analytics for rare targets, according to claim 22, wherein the method further comprises simultaneously updating output feedback to the machine learning models (Fig. 4, Col. 2-3, 67-4, “ Rather than waiting for a model to be trained to be accurate for all of the data in a dataset, the data can be iteratively auto-annotated while the model is still being trained”, Col. 7, 55-59, “Human labels for these objects are fed into the labeled dataset. All previously human-labeled objects are then used to train a new version of the supervised model”, “inference can be performed in batches while iteratively training the machine learning model”, D1, P. 7, “Ekya periodically checkpoints the model (§5) so that inference can get the up-to-date accuracy from retraining”, P. 4, “3 Scheduling retraining and inference jointly We propose joint retraining and inference on edge servers”, P. 9, “Every few epochs, Ekya uses the current accuracy of the model being retrained to estimate its eventual accuracy when all the epochs are complete”).The same motivation to combine for claim 22 equally applies for current claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent Application Publication No. 20200184274 filed by Lee et al. that disclose a technique for improving segmentation performance for new image environments and related diseases by quickly and accurately learning characteristics that are not learned in a medical image on the basis of meta learning and active learning techniques. The learning is balanced using data obtained through re-sampling to reach a balanced learning parameter capable of performing even a newly learned portion without forgetting a well done portion. The learning result can be quickly and accurately converged to the balanced learning parameter by performing meta-learning from the learning sample having the worst learning result.
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148