Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Reopening of Prosecution After Appeal Brief
In view of the appeal brief filed on 07/17/2026, PROSECUTION IS HEREBY REOPENED. New grounds of rejection are set forth below.
To avoid abandonment of the application, appellant must exercise one of the following two options:
(1) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply under 37 CFR 1.113 (if this Office action is final); or,
(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41.31 followed by an appeal brief under 37 CFR 41.37. The previously paid notice of appeal fee and appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth in 37 CFR 41.20 have been increased since they were previously paid, then appellant must pay the difference between the increased fees and the amount previously paid.
A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by signing below:
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4-7, 10, 12, 15-18, 20-24 and 26-29 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
In regards to claim 1,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A – Prong 1: Judicial Exception Recited?
MPEP 2106.04(a)(2)(I) “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.”
Further, the MPEP recites “The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.”
Yes, the claim recites a mental process, specifically:
classifying, based on a numeric value of the data feature meeting a condition, the evaluation data to obtain an evaluation data subset;
This limitation encompasses an evaluation of a given data feature to a given condition and the evaluation data and providing an opinion as an evaluation data subset.
comparing the inference result to a label of the evaluation data to obtain a comparison result indicating whether the inference result and the label are different or the same;
This limitation encompasses an evaluation of the determined inference result and the label to provide a judgement (comparison result) of whether the inference result and the label are different or the same.
calculating, based on the comparison result, an inference accuracy of the Al model to obtain an evaluation result of the Al model on the evaluation data that meet the condition;
This limitation encompasses an evaluation of the comparison result to determine an inference accuracy and providing an opinion of an evaluation result.
generating an optimization suggestion for the AI model based on the evaluation result, wherein the optimization suggestion comprises training the Al model with new data that meet the condition;
This limitation encompasses providing an opinion that the AI model should be trained with new data that meet the condition.
Therefore, the claim recites a mental process.
Step 2A – Prong 2: Integrated into a Practical Solution?
MPEP 2106.05(f) Mere Instructions To Apply An Exception has found simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. The following steps are mere instructions to apply:
A method implemented by a computing device and comprising: (generic computer component to apply the abstract idea)
AI model
determining an inference result of the AI model on the evaluation data; (the BRI of the limitation encompasses merely executing the AI model on the provided data to provide a prediction ie inference result)
and training the Al model with the new data. (wherein training is recited at the highest level of generality)
MPEP 2106.05(g) Insignificant Extra-Solution Activity has found mere data gathering to be insignificant extra-solution activity. The following steps are insignificant extra-solution activities:
Mere data gathering:
obtaining an artificial intelligence (AI) model and an evaluation data set, wherein the evaluation data set comprises evaluation data, and wherein the evaluation data comprise labels indicating a real result corresponding to the evaluation data (obtaining an AI model and an evaluation data set with particular data)
obtaining a data feature of a task type of the AI model, wherein: the evaluation data are images and the data feature is a quantity of bounding boxes, an area ratio of a bounding box to an image, an area variance of the bounding box, a degree of a distance from the bounding box to an image edge, an overlapping degree of the bounding boxes, an aspect ratio of the image, a resolution of the image, a blurriness of the image, or a saturation of the image; the evaluation data are text data and the data feature is a first quantity of words, a second quantity of non-repeated words, a length, a third quantity of stop words, a fourth quantity of punctuations, a fifth quantity of title-style words, a mean length of the words, a term frequency, or an inverse document frequency; or the evaluation data are audio data and the data feature is a short time average zero crossing rate, a short time energy, an entropy of energy, a spectrum centroid, a spectral spread, a spectral entropy, or a spectral flux; (obtaining particular feature of the given AI model)
The additional elements have been considered both individually and as an ordered combination in to determine whether they integrate the exception into a practical application. Therefore, no meaningful limits are imposed on practicing the abstract idea.
The claim is directed to the abstract idea.
Step 2B: Claim provides an Inventive Concept?
No, as discussed with respect to Step 2A, the additional limitation is mere data gathering (Insignificant Extra-Solution Activity) and Mere Instructions To Apply An Exception and a generic device do not impose any meaningful limits on practicing the abstract idea and therefore the claim does not provide an inventive concept in Step 2B.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more.
The recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it".
The claim recites receiving data by generic device.
This has been determined to be insignificant extra-solution activity as found in MPEP § 2106.05(d)(II)(i): Receiving or transmitting data over a network, e.g., using the Internet to
gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary
computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607,
610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP
Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)
(sending messages over a network); buy SAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112
USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network);
but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106
(Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how
interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides
the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink."
(emphasis added)).
The additional elements have been considered both individually and as an ordered
combination in the significantly more consideration.
The claim is ineligible.
In regards to claim 4,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
further comprising obtaining performance data indicating a performance of hardware performing an inference on the evaluation data using the Al model
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
further comprising obtaining performance data indicating a performance of hardware performing an inference on the evaluation data using the Al model
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
This has been determined to be insignificant extra-solution activity as found in MPEP § 2106.05(d)(II)(i): Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
In regards to claim 5,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
further comprising obtaining performance data indicating a usage status of an operator in the Al model while performing an inference on the evaluation data using the AI model
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
further comprising obtaining performance data indicating a usage status of an operator in the Al model while performing an inference on the evaluation data using the AI model
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
This has been determined to be insignificant extra-solution activity as found in MPEP § 2106.05(d)(II)(i): Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
In regards to claim 6,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
wherein the condition comprises sub-conditions, and wherein the evaluation data have data features that correspond to the sub-conditions
This limitation directs to a mathematical relationship wherein recall claim 1 recites “based on a numeric value of the data feature meeting a condition” ie analyzing a numeric value under the condition. See MPEP 2106.04(a)(2)(I)(A.) iv. organizing information and manipulating information through mathematical correlations, Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014). The patentee in Digitech claimed methods of generating first and second data by taking existing information, manipulating the data using mathematical functions, and organizing this information into a new form. The court explained that such claims were directed to an abstract idea because they described a process of organizing information through mathematical correlations, like Flook's method of calculating using a mathematical formula. 758 F.3d at 1350, 111 USPQ2d at 1721.
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
In regards to claim 7,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
wherein each of the data features meets one of the sub- conditions
This limitation directs to a mathematical relationship wherein recall claim 1 recites “based on a numeric value of the data feature meeting a condition” ie analyzing a numeric value under the condition. See MPEP 2106.04(a)(2)(I)(A.) iv. organizing information and manipulating information through mathematical correlations, Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014). The patentee in Digitech claimed methods of generating first and second data by taking existing information, manipulating the data using mathematical functions, and organizing this information into a new form. The court explained that such claims were directed to an abstract idea because they described a process of organizing information through mathematical correlations, like Flook's method of calculating using a mathematical formula. 758 F.3d at 1350, 111 USPQ2d at 1721.
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
In regards to claim 10,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
wherein the usage status comprises a use duration of the operator and a use quantity of the operator
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
wherein the usage status comprises a use duration of the operator and a use quantity of the operator
This limitation directs to mere data gathering of insignificant extra-solution activity. See MPEP § 2106.05(g)
This has been determined to be insignificant extra-solution activity as found in MPEP § 2106.05(d)(II)(i): Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
Claim 12 and 24 are rejected on the same grounds under 35 U.S.C. 101 as claim 1 as they are substantially similar, respectively, Mutatis mutandis.
Claim 15 and 26 are rejected on the same grounds under 35 U.S.C. 101 as claim 4 as they are substantially similar, respectively, Mutatis mutandis.
Claim 16 is rejected on the same grounds under 35 U.S.C. 101 as claim 5 as they are substantially similar, respectively, Mutatis mutandis.
Claim 17 is rejected on the same grounds under 35 U.S.C. 101 as claim 6 as they are substantially similar, respectively, Mutatis mutandis.
Claim 18 is rejected on the same grounds under 35 U.S.C. 101 as claim 7 as they are substantially similar, respectively, Mutatis mutandis.
Claim 20 is rejected on the same grounds under 35 U.S.C. 101 as claim 21 as they are substantially similar, respectively, Mutatis mutandis.
In regards to claim 21,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
further comprising: generating an evaluation report comprising the optimization suggestion and performance data associated with the AI model;
This limitation directs to a mental process that can be performed in the human mind, by a human using pen and paper, or using a computer as a tool to perform the concept and encompasses providing the previous opinion and given performance data. See MPEP 2106.04(a)(2)(III)
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
and sending the evaluation report to a terminal device or a mailbox of a user
This limitation directs to post solution activity of insignificant extra-solution activity. See MPEP § 2106.05(g)
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
and sending the evaluation report to a terminal device or a mailbox of a user
This limitation directs to post solution activity of insignificant extra-solution activity. See MPEP § 2106.05(g)
This has been determined to be insignificant extra-solution activity as found in MPEP § 2106.05(d)(II)(i): Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
In regards to claim 22,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
wherein the evaluation data comprise microbial images, wherein the task type is object detection, and wherein the inference result comprises epithelial cells, blastospores, cocci, white blood cells, spores, fungi, or clue cells
This limitation merely indicates a field of use or technological environment in which the judicial exception is performed. This type of limitation merely confines the use of the abstract idea to a particular technological environment (classification of microbial images) and thus fails to add an inventive concept to the claims. See MPEP § 2106.05(h)
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
wherein the evaluation data comprise microbial images, wherein the task type is object detection, and wherein the inference result comprises epithelial cells, blastospores, cocci, white blood cells, spores, fungi, or clue cells
This limitation merely indicates a field of use or technological environment in which the judicial exception is performed. This type of limitation merely confines the use of the abstract idea to a particular technological environment (classification of microbial images) and thus fails to add an inventive concept to the claims. See MPEP § 2106.05(h)
In regards to claim 23,
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory category. See MPEP 2106.03:
The claim directs to a statutory category – process.
Step 2A Prong 1: The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
wherein the data feature is the aspect ratio, and wherein the condition is that the aspect ratio is even
This limitation directs to a mathematical relationship wherein the aspect ratio being even means that width = height. See MPEP 2106.04(a)(2)(I)(A.)
Step 2A Prong 2: The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B: The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 27 is rejected on the same grounds under 35 U.S.C. 101 as claim 10 as they are substantially similar, respectively, Mutatis mutandis.
Claim 28 is rejected on the same grounds under 35 U.S.C. 101 as claim 22 as they are substantially similar, respectively, Mutatis mutandis.
Claim 29 is rejected on the same grounds under 35 U.S.C. 101 as claim 23 as they are substantially similar, respectively, Mutatis mutandis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-7, 12, 15-18, 23-24, 26 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Pub. No. US20180308202A1 Appu et al. (“Appu”) in view of Liu, Wei, et al. "Ssd: Single shot multibox detector." European conference on computer vision. arXiv:1512.02325v5 [cs.CV] 29 Dec 2016. (“Liu”)
In regards to claim 1 and analogous claims 12 and 24,
Appu teaches A method implemented by a computing device and comprising:
obtaining an artificial intelligence (AI) model and an evaluation data set, wherein the evaluation data set comprises evaluation data,
(Appu, “[0215] A second exemplary type of neural network is the Convolutional Neural Network (CNN) [obtaining an artificial intelligence (AI) model]. A CNN is a specialized feedforward neural network for processing data having a known, grid-like topology, such as image data [an evaluation data set, wherein the evaluation data set comprises evaluation data; wherein the evaluation data is an image]. Accordingly, CNNs are commonly used for compute vision and image recognition applications, but they also may be used for other types of pattern recognition such as speech and language processing. The nodes in the CNN input layer are organized into a set of “filters” (feature detectors inspired by the receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computations for a CNN include applying the convolution mathematical operation to each filter to produce the output of that filter. Convolution is a specialized kind of mathematical operation performed by two functions to produce a third function that is a modified version of one of the two original functions. In convolutional network terminology, the first function to the convolution can be referred to as the input, while the second function can be referred to as the convolution kernel. The output may be referred to as the feature map. For example, the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted by the training process for the neural network.”)
Appu teaches and wherein the evaluation data comprise labels indicating a real result corresponding to the evaluation data;
(Appu, “[0200] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing a problem being modeled by the network, and adjusting the weights until the network model performs with a minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to the input representing an instance in a training data set is compared to the “correct” labeled output for that instance [wherein the evaluation data comprise labels indicating a real result corresponding to the evaluation data], an error signal representing the difference between the output and the labeled output is calculated, and the weights associated with the connections are adjusted to minimize that error as the error signal is backward propagated through the layers of the network. The network is considered “trained” when the errors for each of the outputs generated from the instances of the training data set are minimized.”)
Appu teaches determining an inference result of the AI model on the evaluation data; comparing the inference result to a label of the evaluation data to obtain a comparison result indicating whether the inference result and the label are different or the same;
(Appu, “[0200] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing a problem being modeled by the network, and adjusting the weights until the network model performs with a minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to the input representing an instance in a training data set [determining an inference result of an AI model on the evaluation data; wherein the determined output produced by the network is an inference result of the AI model] is compared to the “correct” labeled output for that instance, an error signal representing the difference between the output and the labeled output is calculated [comparing the inference result to a label of the evaluation data to obtain a comparison result indicating whether the inference result and the label are different or the same; wherein the error signal is the comparison result that indicates whether the output and the label is the same or different], and the weights associated with the connections are adjusted to minimize that error as the error signal is backward propagated through the layers of the network. The network is considered “trained” when the errors for each of the outputs generated from the instances of the training data set are minimized.”)
Appu teaches calculating, based on the comparison result, an inference accuracy of the AI model to obtain an evaluation result of the AI model [on data that meet the condition.]
(Appu, “[0232] Supervised learning is a learning method in which training is performed as a mediated operation, such as when the training dataset 1502 includes input paired with the desired output for the input, or where the training dataset includes input having known output and the output of the neural network is manually graded. The network processes the inputs and compares the resulting outputs against a set of expected or desired outputs. Errors are then propagated back through the system. The training framework 1504 can adjust to adjust the weights that control the untrained neural network 1506. The training framework 1504 can provide tools to monitor how well the untrained neural network 1506 is converging towards a model suitable to generating correct answers based on known input data. The training process occurs repeatedly as the weights of the network are adjusted to refine the output generated by the neural network. The training process can continue until the neural network reaches a statistically desired accuracy [calculating, based on the comparison result, an inference accuracy of the AI model to obtain an evaluation result of the AI model [on data that meet the condition]; wherein the accuracy obtained during training is the inference accuracy] associated with a trained neural net 1508. The trained neural network 1508 can then be deployed to implement any number of machine learning operations.”)
Appu teaches and generating an optimization suggestion for the AI model based on the evaluation result.
(Appu, “[0159] Embodiments provide for a novel technique for adding the ability to configure the processing hardware, such as that of graphics processor 614, application processor 612, etc., to suit a dataset to improve energy efficiency of the inferencing compute [generating an optimization suggestion for the AI model]. For example, inference/prediction data precision may be determined by first detecting and monitoring datasets as facilitated by detection/monitoring logic 701 and simultaneously or subsequently, analyze the precision [based on the evaluation result] associated with such datasets, which, when used and applied, can allow for maintaining energy efficiency while adapting hardware built for superset capabilities.”)
Appu teaches wherein the optimization suggestion comprises training the AI model with new data [that meet the condition]; and training the AI model with the new data.
PNG
media_image1.png
606
577
media_image1.png
Greyscale
(Appu, Figure 15, “[0234] Variations on supervised and unsupervised training may also be employed. Semi-supervised learning is a technique in which in the training dataset 1502 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variant of supervised learning in which input data is continuously used to further train the model. Incremental learning enables the trained neural network 1508 to adapt to the new data 1512 without forgetting the knowledge instilled within the network during initial training [wherein the optimization suggestion comprises training the AI model with new data (see element 1512 of fig. 15)].”
)
However, Appu does not explicitly teach wherein: the evaluation data are images and the data feature is a quantity of bounding boxes, an area ratio of a bounding box to an image, an area variance of the bounding box, a degree of a distance from the bounding box to an image edge, an overlapping degree of the bounding boxes, an aspect ratio of the image, a resolution of the image, a blurriness of the image, or a saturation of the image; the evaluation data are text data and the data feature is a first quantity of words, a second quantity of non-repeated words, a length, a third quantity of stop words, a fourth quantity of punctuations, a fifth quantity of title-style words, a mean length of the words,a term frequency, or an inverse document frequency; or the evaluation data are audio data and the data feature is a short time average zero crossing rate, a short time energy, an entropy of energy, a spectrum centroid, a spectral spread, a spectral entropy, or a spectral flux;
classifying, based on a numeric value of the data feature meeting a condition, the evaluation data to obtain an evaluation data subset
[calculating, based on the comparison result, an inference accuracy of the AI model to obtain an evaluation result of the AI model on] data that meet the condition… new data that meet the condition
Liu teaches obtaining a data feature of a task type of the AI model, wherein: the evaluation data are images and the data feature is a quantity of bounding boxes, an area ratio of a bounding box to an image, an area variance of the bounding box, a degree of a distance from the bounding box to an image edge, an overlapping degree of the bounding boxes, an aspect ratio of the image, a resolution of the image, a blurriness of the image, or a saturation of the image;
the evaluation data are text data and the data feature is a first quantity of words, a second quantity of non-repeated words, a length, a third quantity of stop words, a fourth quantity of punctuations, a fifth quantity of title-style words, a mean length of the words, a term frequency, or an inverse document frequency; or the evaluation data are audio data and the data feature is a short time average zero crossing rate, a short time energy, an entropy of energy, a spectrum centroid, a spectral spread, a spectral entropy, or a spectral flux;
(Liu, “Fig. 1: SSD framework. (a) SSD only needs an input image and ground truth boxes for each object during training. In a convolutional fashion, we evaluate a small set (e.g. 4) of default boxes of different aspect ratios [obtaining a data feature of a task type of the AI model, wherein: the evaluation data are images and the data feature is an aspect ratio of the image] at each location in several feature maps with different scales (e.g. 8 _ 8 and 4 _ 4 in (b) and (c)). For each default box, we predict both the shape offsets and the confidences for all object categories ((c1; c2; … ; cp)).”
PNG
media_image2.png
269
705
media_image2.png
Greyscale
)
Liu teaches classifying, based on a numeric value of the data feature meeting a condition, the evaluation data to obtain an evaluation data subset;
(Liu, Section 2.2, “We impose different aspect ratios for the default boxes, and denote them as ar ∈ {1,2,3, 1/2, 1/3}. We can compute the width (wa k = sk √ar) and height (ha k = sk/√ar) for each default box. For the aspect ratio of 1, we also add a default box whose scale is sk = √sksk+1, resulting in 6 default boxes per feature map location [classifying, based on a numeric value ie aspect ratio of the data feature meeting a condition ie aspect ratio ∈ {1,2,3, 1/2, 1/3}, the evaluation data to obtain an evaluation data subset ie default boxes]. We set the center of each default box to (i+0.5 |fk| , j+0.5 |fk| ), where |fk| is the size of the k-th square feature map, i,j ∈ [0,|fk|). In practice, one can also design a distribution of default boxes to best fit a specific dataset.”)
Liu teaches [calculating, based on the comparison result, an inference accuracy of the AI model to obtain an evaluation result of the AI model on] data that meet the condition… new data that meet the condition
(Liu, Section 2., “Fig. 1: SSD framework. (a) SSD only needs an input image and ground truth boxes for each object during training [new data; wherein the new data may be provided by Appu in the context of the applied references]. In a convolutional fashion, we evaluate a small set (e.g. 4) of default boxes of different aspect ratios [that meet the condition; ie different aspect ratios (see Section 2.2)] at each location in several feature maps with different scales (e.g. 8 _ 8 and 4 _ 4 in (b) and (c)). For each default box, we predict both the shape offsets and the confidences for all object categories ((c1; c2; … ; cp)).”)
Appu and Liu are both considered to be analogous to the claimed invention because they are in the same field of image recognition using artificial intelligence. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu to incorporate the teachings of Liu in order to provide a method of object detection that naturally handle objects at various sizes as doing so provides competitive accuracy to existing methods and better accuracy with smaller input image sizes (Liu, Abstract, “We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines pre dictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into sys tems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For 300×300 in put, SSD achieves 74.3% mAP1 on VOC2007 test at 59 FPS on a Nvidia Titan X and for 512 × 512 input, SSD achieves 76.9% mAP, outperforming a compa rable state-of-the-art Faster R-CNN model. Compared to other single stage meth ods, SSD has much better accuracy even with a smaller input image size.”) (Liu, Section 1., “To achieve high detection accuracy we produce predictions of different scales from feature maps of different scales, and explicitly separate predictions by aspect ratio.”)
In regards to claim 4 and analogous claims 15 and 26,
Appu and Liu teaches The method of claim 1,
Appu teaches further comprising obtaining performance data indicating a performance of hardware performing an inference on the evaluation data using the Al model.
(Appu, Figure 7, “[0185] As illustrated, context scheduler 820 performs monitoring of processor utilization, such as monitoring utilization of graphics processor 614, through GPU utilization monitoring block 825 as facilitated by detection/monitoring logic 701 of FIG. 7. As further illustrated, context-0 and context-1 as represented by EU 831A-831D and EU833A-833F, respectively, have separate address spaces, where, in one embodiment, microcontroller context scheduler 820 of graphics processor 614 monitors how much of graphics processor 614 is utilized. If the utilization is regarded as low, context scheduler 820 may dispatches more contexts, such allowing for solving additional inference issues [obtaining performance data indicating a performance of hardware performing an inference on the evaluation data using the AI model; wherein the performance data ie GPU utilization is monitored while the AI model is performing inferencing such to allow for solving inference issues].”)
In regards to claim 5 and analogous claim 16,
Appu and Liu teaches The method of claim 1,
Appu teaches further comprising obtaining performance data indicating a usage status of an operator in the Al model while performing an inference on the evaluation data using the AI model.
(Appu, “[0181] In the illustrated embodiment, framework 800 includes training data 801, learning block 803, inference data 805, and configurable hardware models 807, where training data 801 is shown as being communicated on to learning block 803 and one or more of configurable hardware models 807, such as communicating configuration information 809 from training data 801 to configurable hardware models 807. Further, in embodiment, receiving inputs from training data 801, learning block 803, and inference data 805, one or more configurable hardware models 807 produce inference/prediction 811, as illustrated.
[0182] For example, to increase the number of operations performed per second, those blocks of processing hardware that are needed for addition, multiplication, accumulation, etc., may be reconfigured using as part of configurable hardware models 807 using configuration information 809 from training data 801 [obtaining performance data indicating a usage status of an operator in the AI model; wherein the usage status of an operator ie the blocks in the model (which can be a layer in the CNN) may be the number of MAC operations performed per second]. This configuration information 809 may be generated at training time [while performing an inference on the evaluation data using the AI model] based on one or more datasets and communicated over to a hardware configuration controller at application and/or graphics processors 612, 614, at runtime, as facilitated by pre-analyzed training logic 703 of FIG. 7.”)
In regards to claim 6 and analogous claim 17,
Appu and Liu teaches The method of claim 1,
Liu teaches wherein the condition comprises sub-conditions, and wherein the evaluation data have data features that correspond to the sub-conditions.
(Liu, Section 2.2, “We impose different aspect ratios for the default boxes, and denote them as ar ∈ {1,2,3, 1/2, 1/3} [wherein the condition ie the aspect ratio being in the fixed set comprises sub-conditions ie each value in {1,2,3, 1/2, 1/3}, and wherein the evaluation data have data features that correspond to the sub-conditions]. We can compute the width (wa k = sk √ar) and height (ha k = sk/√ar) for each default box. For the aspect ratio of 1, we also add a default box whose scale is sk = √sksk+1, resulting in 6 default boxes per feature map location. We set the center of each default box to (i+0.5 |fk| , j+0.5 |fk| ), where |fk| is the size of the k-th square feature map, i,j ∈ [0,|fk|). In practice, one can also design a distribution of default boxes to best fit a specific dataset.”)
In regards to claim 7 and analogous claim 18,
Appu and Liu teaches The method of claim 6,
Liu teaches wherein each of the data features meets one of the sub- conditions.
(Liu, Section 2.2, “We impose different aspect ratios for the default boxes, and denote them as ar ∈ {1,2,3, 1/2, 1/3} [wherein each of the data features meets one of the sub- conditions]. We can compute the width (wa k = sk √ar) and height (ha k = sk/√ar) for each default box. For the aspect ratio of 1, we also add a default box whose scale is sk = √sksk+1, resulting in 6 default boxes per feature map location. We set the center of each default box to (i+0.5 |fk| , j+0.5 |fk| ), where |fk| is the size of the k-th square feature map, i,j ∈ [0,|fk|). In practice, one can also design a distribution of default boxes to best fit a specific dataset.”)
In regards to claim 23 and analogous claim 29,
Appu and Liu teaches The method of claim 1,
Liu teaches wherein the data feature is the aspect ratio, and wherein the condition is that the aspect ratio is even.
(Liu, Section 2.2, “We impose different aspect ratios for the default boxes, and denote them as ar ∈ {1,2,3, 1/2, 1/3} [wherein the data feature is the aspect ratio]. We can compute the width (wa k = sk √ar) and height (ha k = sk/√ar) for each default box. For the aspect ratio of 1, we also add a default box whose scale is sk = √sksk+1, resulting in 6 default boxes per feature map location. We set the center of each default box to (i+0.5 |fk| , j+0.5 |fk| ), where |fk| is the size of the k-th square feature map, i,j ∈ [0,|fk|). In practice, one can also design a distribution of default boxes to best fit a specific dataset [and wherein the condition is that the aspect ratio is even; ex aspect ratio ∈ {2, 1/2}].”)
Appu and Liu are both considered to be analogous to the claimed invention because they are in the same field of image recognition using artificial intelligence. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu and Liu to incorporate the teachings of Liu in order to provide the condition that the aspect ratio is even as doing so allows a distribution of default boxes to best fit a specific dataset (Liu, Section 2.2, “In practice, one can also design a distribution of default boxes to best fit a specific dataset.”)
Claim(s) 10 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Appu in view of Liu in further view of US Pub No. US20160358099A1 Sturlaugson et al. (“Sturlaugson”)
In regards to claim 10 and analogous claim 27,
Appu and Liu teaches The method of claim 5,
However, Appu and Liu does not explicitly teach wherein the usage status comprises a use duration of the operator and a use quantity of the operator.
Sturlaugson teaches wherein the usage status comprises a use duration of the operator and a use quantity of the operator.
Examiner’s note: Examiner interprets the limitation in light of the specification if the instant application ([0092], “The use quantity of the operator is a quantity of times that the operator is used in the process in which the inference module 503 performs inference. The use duration of the operator is total duration and/or average duration used by each operator in the process in which the inference module 503 performs inference.”)
(Sturlaugson, “[0042] The performance result for each machine learning model 32 and/or the individual evaluation results for each round of validation may include an indicator, value, and/or result related to a correlation coefficient, a mean square error, a confidence interval, an accuracy, a number of true positives, a number of true negatives, a number of false positives, a number of false negatives, a sensitivity, a positive predictive value, a specificity, a negative predictive value, a false positive rate, a false discovery rate, a false negative rate, and/or a false omission rate. Additionally or alternatively, the indicator, value, and/or result may be related to computational efficiency [a use quantity of the operator; ie a quantity of times an operator is used as a model of computational efficiency], memory required, and/or execution speed [usage status comprises a use duration of the operator; ie total duration used by each operator in the process as a model of execution speed].”)
Sturlaugson is considered to be analogous to the claimed invention because they are in the same field of performance monitoring of artificial intelligence systems. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu and Liu to incorporate the teachings of Sturlaugson in order to provide a method of evaluating trained model performance as doing so allows evaluating models for applicability to various specific problems (Sturlaugson, “[0004] The large number of machine learning options available to address a problem makes it difficult to choose the best option or even a well-performing option. The amount, type, and quality of data affect the accuracy and stability of training and the resultant trained models. Further, problem-specific considerations, such as tolerance of errors (e.g., false positives, false negatives) scalability, and execution speed, limit the acceptable choices. [0005] Therefore, there exists a need for comparing machine learning models for applicability to various specific problems.”)
Claim(s) 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Appu in view of Liu in further view of Sturlaugson in further view of Wexler, James, et al. "The what-if tool: Interactive probing of machine learning models." arXiv:1907.04135v1 [cs.LG] 9 Jul 2019. (“Wexler”)
In regards to claim 21 and analogous claim 20,
Appu and Liu teaches The method of claim 1,
However, Appu and Liu does not explicitly teach further comprising: generating an evaluation report comprising the optimization suggestion and performance data associated with the AI model; and sending the evaluation report to a terminal device or a mailbox of a user
Sturlaugson teaches further comprising: generating an evaluation report comprising [the optimization suggestion] and performance data associated with the AI model; and sending the evaluation report to a terminal device or a mailbox of a user.
(Sturlaugson, “[0046] Machine learning systems 10 may include an optional presentation module 44 that is configured to present the performance comparison statistics [performance data associated with the AI model] to an operator and/or a user of the machine learning system 10. The presentation module 44 may be configured to present the performance results for all of the machine learning models in a unified format to facilitate comparison of the machine learning models 32. The presentation module 44 may be configured to display the performance comparison statistics by visual, audio, and/or tactile display. Displays may include an alphanumeric display, a video monitor, a lamp, an LED, a speaker, a buzzer, a spring, and/or a weight. Additionally or alternatively, presentation module 44 may store a file [generating an evaluation report] including the performance comparison statistics in the persistent storage 18 and/or transmit a data block including the performance comparison statistics to the storage unit 14 and/or a user [and sending the evaluation report to a terminal device or a mailbox of a user].”)
Sturlaugson is considered to be analogous to the claimed invention because they are in the same field of performance monitoring of artificial intelligence systems. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu and Liu to incorporate the teachings of Sturlaugson in order to provide a method of evaluating trained model performance as doing so allows evaluating models for applicability to various specific problems (Sturlaugson, “[0004] The large number of machine learning options available to address a problem makes it difficult to choose the best option or even a well-performing option. The amount, type, and quality of data affect the accuracy and stability of training and the resultant trained models. Further, problem-specific considerations, such as tolerance of errors (e.g., false positives, false negatives) scalability, and execution speed, limit the acceptable choices. [0005] Therefore, there exists a need for comparing machine learning models for applicability to various specific problems.”)
However, Sturlaugson does not explicitly teach [generating an evaluation report] comprising the optimization suggestion
Wexler teaches [generating an evaluation report] comprising the optimization suggestion (recall the optimization suggestion is training the AI model with new data that meet the condition)
(Wexler, Section 1., “We present the What-If Tool (WIT), an open-source, model-agnostic interactive visual tool for model understanding. Running the tool requires only trained models and a sample dataset [comprising the optimization suggestion; wherein WIT allows users to explore the impacts of new data on trained models]. The tool is available as part of TensorBoard, the front-end for the TensorFlow [5] machine learning framework; it can also be used in Jupyter and Colaboratory notebooks.
WIT supports iterative “what-if” exploration through a visual interface. Users can perform counterfactual reasoning, investigate decision boundaries, and explore how general changes to data points affect predictions, simulating different realities to see how a model behaves. It supports flexible visualizations of input data and of model performance, letting the user easily switch between different views.
An important motivation for these visualizations is to enable easy slicing of the data by a combination of different features from the dataset. We refer to this as intersectional analysis, a term we chose in reference to Crenshaw's concept of “intersectionality” [10]. This type of analysis is critical for understanding issues surrounding model fairness investigations [9].
The tool supports both local (analyzing the decision on a single data point) and global (understanding model behavior across an entire dataset) model understanding tasks [11]. It also supports a variety of data and model types.”)
Wexler is considered to be analogous to the claimed invention because they are in the same field of performance monitoring of artificial intelligence systems. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu and Liu and Sturlaugson to incorporate the teachings of Wexler in order to provide a tool for understanding model performance across a wide range of inputs as doing so allows practitioners to test performance in hypothetical situations, analyze importance of data features, and visualize model behavior across multiples models and subsets of input data (Wexler, Abstract, “A key challenge in developing and deploying Machine Learning (ML) systems is understanding their performance across a wide range of inputs. To address this challenge, we created the What-If Tool, an open-source application that allows practitioners to probe, visualize, and analyze ML systems, with minimal coding. The What-If Tool lets practitioners test performance in hypothetical situations, analyze the importance of different data features, and visualize model behavior across multiple models and subsets of input data. It also lets practitioners measure systems according to multiple ML fairness metrics. We describe the design of the tool, and report on real-life usage at different organizations.”)
Claim(s) 22 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Appu in view of Liu in further view of Hao, X. Wang, J. Zhang, J. Liu, X. Du and L. Liu, "Automatic Detection of Fungi in Microscopic Leucorrhea Images Based on Convolutional Neural Network and Morphological Method," 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) (“Hao”)
In regards to claim 22 and analogous claim 28,
Appu and Liu teaches The method of claim 1,
Hao teaches wherein the evaluation data comprise microbial images, wherein the task type is object detection, and wherein the inference result comprises epithelial cells, blastospores, cocci, white blood cells, spores, fungi, or clue cells.
(Hao, Section I, “In this paper, we propose an automatic identification [wherein the task type is object detection] of fungi in microscopic leucorrhea images [evaluation data comprise microbial images] based on convolutional neural network and morphological method. The structure is organized as follows: Section 2 used the maximum inter-class variance method to segment original image and obtained possible fungi subimages. Section 3 establishes a fully trained CNN to recognize fungi [wherein the inference result comprises epithelial cells, blastospores, cocci, white blood cells, spores, fungi, or clue cells]. Section 4 presents morphological method to further classify the selected candidate. Section 5 shows the experimental results. Finally, Section 6 concludes the paper and discusses the future work.”)
Hao is considered to be analogous to the claimed invention because they are in the same field of convolutional neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Appu and Liu to incorporate the teachings of Hao in order to apply the methods of Appu and Liu to improve detection accuracy of fungi and provide important evidence for fungal vaginitis (Hao, Abstract, “Leucorrhea routine test is one of the most widely used tests in gynecological examinations, and fungi inspection is vital for gynecological test because fungi is an important evidence for fungal vaginitis. In order to improve detection accuracy, an automatic identification of fungi in microscopic leucorrhea images based on convolutional neural network (CNN) and morphological method is proposed in this paper. First, we use the maximum inter-class variance method to segment original image and obtain possible fungi subimages. Then, a fully trained CNN is applied to recognize fungi. Finally, morphological method, such as template match method and concave point detection method, is used to further classify the selected candidate to improve recognize accuracy. In experiments, the method using CNN and morphological method achieved 93.26% accuracy.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL: G. Campanile, C. Di Ruberto and A. Loddo, "An Open Source Plugin for Image Analysis in Biology," 2019 IEEE 28th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), Napoli, Italy, 2019, pp. 162-167, doi: 10.1109/WETICE.2019.00042 (“Campanile”) discloses aspect ratio as a morphological feature for image analysis in biology.
US Pub no. US20180336493A1 Hayes et al. (“Hayes”) discloses generating optimal suggestions of machine learning models in response to API requests.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights