DETAILED ACTION
Response to Amendment
The amendment filed on 6 April 2026 has been entered.
Claims 1-21 are pending.
Claims 1, 8, 15 are amended.
Response to Arguments
Applicant’s remarks, regarding the rejections of claims under 35 USC 101, have been fully considered.
Applicant submits the claimed operations have been repeatedly mischaracterized in the Office Action as constituting "mental processes" by contending, for example:
Ignored labeling target: "identifying... an unlabeled vector... as an ignored labeling target... (mental process of judgment)" (on page 3 of the Final OA dated 5/16/2025 and on page 11 of the current OA dated 12/05/2025)
Angle calculations: "calculating a first angle... and a second angle... (mental process of evaluation)" (page 3 of the Final OA dated 5/16/2025 and page 11 of the current OA dated 12/05/2025)
Search for examples: "search for examples... (mental process of judgment)" (page 5 of the current OA dated 12/05/25)
Further, Applicant submits the following arguments.
1. The Claimed Vector-Space Angle Calculations Are Not Practically Performable In The Human Mind
2. Identifying Orthogonality To A Space Spanned By Multiple Labeled Vectors Is Not A Mental Process
3. Training A Model Using Newly Generated Labeled Data Cannot Be Performed Mentally
Examiner respectfully disagrees.
As outlined in the Non-Final Office Action mailed 5 December 2025 and below, Claims 1, 8, 15 recites a judicial exception (abstract idea) under Step 2A Prong One.
Regarding Applicant’s arguments that claimed vector-space angle calculations are not practically performable in the human mind and identifying orthogonality to a space spanned by multiple labeled vectors is not a mental process, Examiner submits arguments pertain to claim limitations which directly recite abstract ideas of mental judgement and evaluation involving mathematical functions, because claim(s) recites a limitation that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper, see MPEP 2106.04(a)(2)(III)(B). Examiner submits application of orthogonal heuristic to calculate angles involving vectors represents a mathematical formula, reciting an abstract idea, see MPEP 2106.04(II)(A)(1). Examiner notes the training of the mathematical model was determined to be directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception. Examiner submits claims can recite a mental process even if they are claimed as being performed on a computer and Claims 1, 8, 15, as claimed in light of the Specification, recite limitations directed to performing the mental processes on a generic computer and using a computer as a tool to perform the mental processes, such as performing “human cognitive actions” and steps recited at a high level of generality and merely used computers as a tool to perform the processes, respectively, see MPEP 2106.04(a)(2)(III)(C).
Applicant submits, regarding Step 2A Prong Two,
The Claimed Technological Improvement Is Improperly Classified As A Field-Of-Use Limitation
The Claims Integrate The Alleged Abstract Idea Into A Practical Application Under Step 2A Prong Two
C. The Step 2B Analysis Is Conclusory and Unsupported by Evidence, Contrary to Berkheimer
Applicant submits the claims do not recite a mental process under Step 2A Prong One and they integrate any alleged abstract idea into a practical application under Step 2A Prong Two by reciting a specific, technologically grounded improvement to machine-learning model training.
Regarding 1. The Claimed Technological Improvement Is Improperly Classified As A Field-Of-Use Limitation, Applicant submits the claims recite a specific improvement to model-training workflows: generating a new labeled vector based on computed angular relationships, identifying unlabeled vectors that are orthogonal to the labeled manifold and training a model using the expanded labeled set. These are functional, state-changing operations that define the mechanism by which the model's training data and resulting behavior are improved. They are not "extra-solution activity" or field-of-use limitations; they are the solution itself.
In response to Applicant’s argument that the claims integrate the abstract ideas into a practical application because the claims improve a technical field, Examiner submits employing generic computer functions, outlined in the Non-Final Office Action mailed 5 December 2025, to execute an abstract idea, even when limiting the use of the idea (such as the mental judgements and evaluations involving mathematical functions presented) to one particular environment, does not add significantly more, similar to how limiting the abstract idea in Flook to petrochemical and oil-refining industries was insufficient, MPEP 2106.05(h).
Examiner submits limitations reciting mere data gathering, outlined in the Non-Final Office Action mailed 5 December 2025, are considered examples of activities found to be insignificant extra-solution activity, and when added to the judicial exception, does not meaningfully limit the claim, see MPEP 2106.05(g).
Further, the judicial exception alone cannot provide the improvement, MPEP 2106.05(a).
Finally, generically “training, by the processing resource, a mathematical model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector”, as claimed, does not provide specific steps and details for technical benefits of mathematical model training or specific improvement to model-training workflows. The claims do not include specific details or steps of how to apply the judicial exception in order to implement and improve the training of the mathematical model or improve model-training workflows as presented by Applicant, MPEP 2106.04(d)(III).
The Examiner further notes, that additional claim elements of the claimed invention are considered insufficient to transform a judicial exception to a patentable invention. The limitations of the claimed inventions do not appear to recite steps for a specific solution to a problem in an existing technology area, where the Applicant’s Specification has set forth an improvement in technology in a non-conclusory manner. Under Step2B, Claims 1, 8, 15 as a whole, do not amount to significantly more than the exception itself (there is no inventive concept in the claim), see MPEP § 2106.05(B)(II).
The rejections of Claims 1-21, under 35 USC 101, have been maintained.
Applicant’s remarks, regarding the rejections of claims under 35 USC 103, have been fully considered.
Applicant submits Tu performs the opposite operation: it removes problematic labeled samples rather than identifying and labeling previously ignored unlabeled vectors orthogonal to the labeled manifold. Tu therefore teaches away from the presently recited operation and cannot, as a factual matter, supply the missing limitation. Since the primary references fail to disclose the claimed orthogonality-based identification technique and Tu addresses a different problem using a contrary mechanism, the cited combination cannot be relied upon for disclosing for the recited limitation. Examiner's construction requires treating already-labeled, already-processed samples as "ignored labeling targets" - the precise opposite of what the Specification defines as unlabeled, orthogonal, previously unexplored vectors - it cannot qualify as the broadest reasonable interpretation. A construction that directly contradicts the Specification is legally unreasonable and the rejection cannot stand on such a foundation.
Examiner respectfully disagrees.
Examiner submits Tu teaches limitations of Claims 1, 8, 15, outlined in the Non-Final Office Action mailed 5 December 2025. Examiner notes Applicant arguments regarding Tu have been addressed in the Non-Final Office Action mailed 5 December 2025 and Examiner’s response to arguments regarding Tu is maintained. Further, Examiner notes “likely to be ignored”, “may yield value”, (i.e., ignored labeling targets), as outlined in the Specification [000137] can be interpreted to be resulting ignored labeling targets that are the product of the data vectors that are ignored after analysis, as broadest reasonable interpretation of ignored is interpreted in the past tense of actions, events, or states that have already occurred and are completed and value yielded may be attributed to any characteristics. Applicant’s argument that Tu “removes problematic labeled samples rather than identifying and labeling previously ignored unlabeled vectors orthogonal to the labeled manifold” pertains to limitations which are not claimed. Claim limitations recite ignored labeling target, not previously ignored. Finally, Examiner notes "Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim. For example, a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment.", see MPEP 2111.01(II).
The rejections of Claims 1-21, under 35 USC 103, have been maintained.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception, abstract idea, without significantly more.
Step 1: This part of the eligibility analysis evaluates whether the claim(s) falls within any statutory
category. MPEP 2106.03:
According to the first part of the Alice analysis, in the instant case, the claims were determined
to be directed to one of the four statutory categories: an article of manufacture, a method/process (Claims 1-7), a machine/system/product (Claims 8-21), and a composition of matter. Based on the claims being determined to be within of the four categories (i.e., process, machine, manufacture, or composition of matter), (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea).
Step 2A Prong One: This part of the eligibility analysis evaluates whether the claim(s) recites a
judicial exception.
Regarding independent claims 1, 8, 15, the claims recite a judicial exception (i.e., an abstract idea enumerated in the 2019 PEG) without significantly more (Step-2A: Prong One). The applicant's claim limitations under broadest reasonable interpretation covers activities classified under mental processes - concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection Ill) and the 2019 PEG. As evaluated below:
Claims 1, 8:
“identifying, by the processing resource, an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target that is orthogonal to a space spanned by the plurality of labeled vectors by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors” (mental process of judgement)
“calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector” (mental process of evaluation)
If the identified limitation(s) falls within at least one of the groupings of abstract ideas, it is
reasonable to conclude that the claim(s) recites an abstract idea in Step 2A Prong One.
Step 2A Prong Two: This part of the eligibility analysis evaluates whether the claim(s) as a whole integrates the recited judicial exception into a practical application of the exception. As evaluated below:
“receiving, by a processing resource, a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector”
“creating, by the processing resource, a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions for mere data gathering or data output, see MPEP 2106.05(g).
“training, by the processing resource, a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception, see MPEP 2106.05(h).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea when considered as an ordered combination and as a whole.
Step 2B: This part of the eligibility analysis evaluates whether the claim, as a whole, amounts to
significantly more than the recited exception, i.e., whether any additional element, or combination of
additional elements, adds an inventive concept to the claim. MPEP 2106.05.
First, the additional elements considered as part of the preamble and the additional elements
directed to the use of computer technology are deemed insufficient to transform the judicial exception
to a patentable invention to a patentable invention because they generally link the judicial exception to
the technology environment, see MPEP 2106.05(h).
Second, the additional elements mere application of the abstract idea or mere instructions to
implement an abstract idea on a computer are deemed insufficient to transform the judicial exception
to a patentable invention to a patentable invention because the limitations generally apply the use of a
generic computer and/or process with the judicial exception, see MPEP 2106.05(f).
Lastly, the claims are directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception. The courts have found these types of limitations insufficient to transform the judicial exception to a patentable invention, see MPEP 2106.05(g).
Furthermore, when considering evidence in view of Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018), see USPTO Berkheimer Memorandum (April 2018). Examiner notes Berkheimer: Option 2 - A citation to one or more of the court decisions discussed in MPEP § 2106.0S(d}(II} as noting the well understood, routine, conventional nature of the additional element (s) (e.g., limitations directed to mere data gathering):
The courts have recognized the following computer functions as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity, see MPEP 2106.05(d).
The additional limitations, as analyzed, failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above. Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible. Therefore, in examining elements as recited by the limitations individually and as an ordered combination, as a whole, claims 1, 8 do not recite what the courts have identified as "significantly more".
Claim 15:
“search for examples that are orthogonal to a problem space spanned by the plurality of labeled vectors that has not vet been explored to identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors” (mental process of judgement)
“calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector” (mental process of evaluation)
If the identified limitation(s) falls within at least one of the groupings of abstract ideas, it is
reasonable to conclude that the claim(s) recites an abstract idea in Step 2A Prong One.
Step 2A Prong Two: This part of the eligibility analysis evaluates whether the claim(s) as a whole integrates the recited judicial exception into a practical application of the exception. As evaluated below:
“receive a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector”
“create a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions for mere data gathering or data output, see MPEP 2106.05(g).
“train a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception, see MPEP 2106.05(h).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea when considered as an ordered combination and as a whole.
Step 2B: This part of the eligibility analysis evaluates whether the claim, as a whole, amounts to
significantly more than the recited exception, i.e., whether any additional element, or combination of
additional elements, adds an inventive concept to the claim. MPEP 2106.05.
First, the additional elements considered as part of the preamble and the additional elements
directed to the use of computer technology are deemed insufficient to transform the judicial exception
to a patentable invention to a patentable invention because they generally link the judicial exception to
the technology environment, see MPEP 2106.05(h).
Second, the additional elements mere application of the abstract idea or mere instructions to
implement an abstract idea on a computer are deemed insufficient to transform the judicial exception
to a patentable invention to a patentable invention because the limitations generally apply the use of a
generic computer and/or process with the judicial exception, see MPEP 2106.05(f).
Lastly, the claims are directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception. The courts have found these types of limitations insufficient to transform the judicial exception to a patentable invention, see MPEP 2106.05(g).
Furthermore, when considering evidence in view of Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018), see USPTO Berkheimer Memorandum (April 2018). Examiner notes Berkheimer: Option 2 - A citation to one or more of the court decisions discussed in MPEP § 2106.0S(d}(II} as noting the well understood, routine, conventional nature of the additional element (s) (e.g., limitations directed to mere data gathering):
The courts have recognized the following computer functions as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity, see MPEP 2106.05(d).
The additional limitations, as analyzed, failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above. Thus, considering the additional elements individually and in combination and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible. Therefore, in examining elements as recited by the limitations individually and as an ordered combination, as a whole, claim 15 does not recite what the courts have identified as "significantly more".
Furthermore, regarding dependent claims 2-7, which depend from claim 1, claims 9-14, which depend from claim 8, and claims 16-21, which depend from claim 15, the claims are directed to a judicial exception (i.e., an abstract idea enumerated in the 2019 PEG, a law of nature, or a natural phenomenon) without significantly more as highlighted below in the claim limitations by evaluating the claim limitations under the Step2A and 2B:
Claims 2, 9, 16:
Incorporates the rejection of claims 1, 8, 15, respectively.
“determining, by the processing resource, that the first angle is less than the second angle” (mental process of judgement)
“identifying, by the processing resource, the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle” (mental process of judgement)
The recitation is directed to mere instructions to implement an abstract idea on a computer, or
merely uses a computer as a tool to perform an abstract idea and are considered to adding the words "apply it" (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to mere instructions to implement an abstract idea on a computer/using computer as a tool cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 3, 10, 17:
Incorporates the rejection of claims 2, 9, 16, respectively.
“comparing, by the processing resource, the minimum angle with a threshold value” (mental process of judgement)
The recitation is directed to mere instructions to implement an abstract idea on a computer, or
merely uses a computer as a tool to perform an abstract idea and are considered to adding the words "apply it" (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to mere instructions to implement an abstract idea on a computer/using computer as a tool cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 4, 11, 18:
Incorporates the rejection of claims 3, 10, 17, respectively.
“identifying, by the processing resource, the unlabeled vector as a high value labeling target where the minimum angle exceeds the threshold value” (mental process of judgement)
The recitation is directed to mere instructions to implement an abstract idea on a computer, or
merely uses a computer as a tool to perform an abstract idea and are considered to adding the words "apply it" (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to mere instructions to implement an abstract idea on a computer/using computer as a tool cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 5, 12, 19:
Incorporates the rejection of claims 3, 10, 17, respectively.
“threshold value is user programmable”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception, see MPEP 2106.05(h).
Limitations directed to mere instructions indicating a field of use or technological environment in which to apply a judicial exception cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 6, 13, 20:
Incorporates the rejection of claims 1, 8, 15, respectively.
“using, by the processing resource, the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors” (mental process of judgement)
The recitation is directed to mere instructions to implement an abstract idea on a computer, or
merely uses a computer as a tool to perform an abstract idea and are considered to adding the words "apply it" (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to mere instructions to implement an abstract idea on a computer/using computer as a tool cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claims 7, 14, 21:
Incorporates the rejection of claims 6, 13, 20, respectively.
“wherein the at least one other heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic”
These recitations are deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to instructions merely indicating a field of use or technological environment in which to apply a judicial exception, see MPEP 2106.05(h).
Limitations directed to mere instructions indicating a field of use or technological environment in which to apply a judicial exception cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
The dependent claims as analyzed above, do not recite limitations that integrated the judicial exception into a practical application. In addition, the claim limitations do not include additional elements that are sufficient to amount to significantly more than the judicial exception (Step-2B). Therefore, the claims do not recite any limitations, when considered individually or as a whole, that recite what have the courts have identified as "significantly more", see MPEP 2106.05; and therefore, as a whole the claims are not patent eligible. As shown above, the dependent claims do not provide any additional elements that when considered individually or as an ordered combination, amount to significantly more than the abstract idea identified. Therefore, as a whole, the dependent claims do not recite what have the courts have identified as "significantly more" than the recited judicial exception. Therefore, claims 2-7, 9-14, and 16-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception and does not recite, when claim elements are examined individually and as a whole, elements that the courts have identified as "significantly more" than the recited judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (U.S. Pre-Grant Publication No. 2021/0150340, hereinafter ‘Liu'), in view of Ge et al. (NPL: "Classification using hyperdimensional computing: A review", hereinafter 'Ge'), Hao et al. (NPL: "HSME: Hypersphere manifold embedding for visible thermal person re-identification", hereinafter 'Hao'), and further in view of Tu et al. (NPL: "Density Peak-Based Noisy Label Detection for Hyperspectral Image Classification", hereinafter 'Tu').
Regarding claim 1, Liu teaches A method comprising: receiving, by a processing resource, a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector ([0027] An embodiment described herein provides that a receiving, by a processing resource, a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector small dataset may be used for the neural network to learn characteristics of the radius of the input vector to the origin. In this way, an OOD vector may be identified when the OOD vector is sufficiently close to the origin (identified through learning), or when the OOD vector is orthogonal to all reference class vectors.; [0075] Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a pre-defined threshold distance, the input sample x may be determined to be OOD.);
Liu fails to teach identifying, by the processing resource, an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target that is orthogonal to a space spanned by the plurality of labeled vectors by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector; creating, by the processing resource, a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector; and training, by the processing resource, a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector.
Ge teaches calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector ([C. Similarity Measurement, pg. 32] For non-binary hypervectors, cosine similarity, defined by Eq. (1), is used to measure their similarity, focusing on the angle and ignoring the impact of the magnitude of hypervectors, where · denotes the magnitude. Unlike the inner product operation [12] of two vectors that affects magnitude and orientation, the cosine similarity only depends on the orientation. In most HD algorithms with non-binary hypervectors, cosine similarity is more often used than inner product. Moreover, when cos( , A B) is close to 1, this implies an extremely high level of similarity. For example, cos( , A B) = 1 indicates calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector two hypervectors A and B are identical. When they are at right angle, then cos( , A B) , = 0 and the two orthogonal vectors are considered dissimilar.; [(7) HD Computing for Semi-Supervised Learning, pg. 42] In [53], SemiHD has been proposed as a self-training or self-learning approach for semi-supervised learning, where the training data is composed of a small portion of labeled data and a large portion of unlabeled data. The SemiHD framework is depicted in Fig. 13 and the flow is illustrated as follows. 1). Encode all the data points, labeled and unlabeled, into HD space with d =10,000 dimensions. 2). Start training from the labeled data to generate k hypervectors, each representing one class. 3). Predict the label for unlabeled data points. Labeling is performed by checking the similarity of unlabeled data with all the class hypervectors, and return the label which shows the highest similarity.);
Liu and Ge are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Ge to Liu before the effective filing date of the claimed invention in order to perform computing which is robust, scalable, energy efficient and requires less time for training and inference (cf. Ge, [1. Introduction, pg. 31] As a brain-inspired computing model, HD computing is robust, scalable, energy efficient and requires less time for training and inference [9]. These features are a result of its ultra-wide data representation and underlying mathematical operations. One thing that should be emphasized is the concept of orthogonality of the hypervectors.).
Hao teaches creating, by the processing resource, a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine a labeling value of the unlabeled vector black arrow, resulting in the anchor creating, by the processing resource, a newly labeled vector classified into class 1 or class 2 dependent on the combination of said angles); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu, Ge, and Hao are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu and Ge, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Hao to Liu before the effective filing date of the claimed invention in order to achieve better performance in matching features (cf. Hao, [1. Introduction, pg. 8386] We also propose a novel two-stage training scheme. In the first stage, the dual-stream network is trained with randomly initialized weights. In second stage, the weight matrix of Sphere Softmax is decomposed into three parts by Singular Value Decomposition(SVD). We use the product of left-unitary matrix and singular value matrix to replace the previous weight matrix as new weight matrix. Then we train the network with fixed Sphere Softmax weight matrix. As the left-unitary is orthogonal and singular value matrix is diagonal matrix, the new weight matrix is also orthogonal. Because of the orthogonal weight matrix, the deep feature representations of different person are relatively independent. Thus, the decorrelated features can achieve better performance in matching problem.).
Tu teaches identifying, by the processing resource, an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target that is orthogonal to a space spanned by the plurality of labeled vectors by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including ([A. Calculating the Distances of the Training Samples, pg. 1575] Let x = {x1, x2,..., xM } refers to the original training set, in which M is the number of classes, and xm refers to the training samples in the mth class. For that is orthogonal to a space spanned by the plurality of labeled vectors two training samples belonging to the mth class, i.e., x j a and x j b, the distance dm ab between two samples can be measured. In this paper, four types of distances, i.e., the ED [44], orthogonal projection divergence (OPD) [45], spectral information divergence (SID) [46], and CC [20], are considered. The definitions of these distance metrics are presented in the following. 1) Euclidean Distance: dj ab = x j a − x j b 2 2. (6) by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including 2) Orthogonal Projection Divergence: dj ab = x j a T Wax j a + x j b T Wbx j b (7) where Wa = 1 − x j a(x j a T x j a)−1x j a T and Wb = 1 − x j b(x j b T x j b)−1x j b T . 3) Spectral Information Divergence: dj ab = u log u v +v log v u (8) where u = (x j a/ x j a) and v = (x j b/ x j b) refer to the desired probability vectors resulting from the pixel vectors x j a and x j b. 4) Correlation Coefficient: dj ab = covx j a, x j b varx j a · varx j b (9) where var(x j a) and var(x j b) refer to variances of the pixel vectors x j a and x j b.; [B. Calculating the Local Densities of the Training Samples, pg. 1576] First, the cutoff distance dm c is calculated as follows: dm c = Sm(t) s.t. t = Nm · (Nm − 1) 100 · p (11) where Sm is a matrix that sorts the nonzero elements in the upper triangular matrix of Dm from the smallest to the largest, p is a free parameter that will be analyzed in Section IV-B, and < · > refers to the round operation. With the above-obtained cutoff distance, the local densities ρm = {ρm 1 , ρm 2 ,..., ρm Nm } of the pixels in the mth class can be calculated as follows: ρm = e − Dm dm c 2 . (12); [C. Detecting the Mislabeled Samples, pg. 1576] Once the local densities of the training samples in different classes are obtained, the identifying, by the processing resource, an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target mislabeled samples can be easily detected and removed as follows: Ym i = Xm i if ρm Nm ≥ λ · ρm ∅ Otherwise (13) where Y = {Y1, Y2,..., YM } refers to the resulting training set, in which the noisy labels have been detected and removed. λ is a free parameter.; Tu teaches using Euclidean Distance, Orthogonal Projection Divergence, Spectral Information Divergence, and Correlation Coefficient to calculate distances between sample vectors in classes and using said distances to determine density in sample data clusters to detect mislabeled samples (labeling target), removing noisy labels resulting in robust detection of mislabeled samples.)
training, by the processing resource, a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector ([IV. EXPERIMENTS, pg. 1576-1577] This paper adopts the SVM classifier to demonstrate the effectiveness of the based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector proposed noisy label detection method as it is one of the most widely used pixelwise classifiers. The SVM method is implemented using the LIBSVM library [47], and the parameters of the SVM are decided using a fivefold cross validation. Furthermore, in all experiments, three widely used quality indexes, i.e., the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient, are used to evaluate the performance of the proposed method. OA measures the percentage of all correctly classified pixels. AA represents the average value of the percentage of the correctly classified pixels for each class. The Kappa coefficient estimates the percentage of classified pixels corrected by the number of agreements that would be expected purely by chance. All experiments are repeated 10 times with randomly selected training samples so as to obtain the mean and standard variances of OA, AA and Kappa. The training sets are constructed using the samples in the ground truth. For each class, some pixels randomly selected from other classes will be added so as to simulate the “noisy label” problem.; [E. Performance Evaluation Using the SVM, pg. 1578-1579] It can be seen that the proposed method can training, by the processing resource, a machine-learning model improve the classification accuracies for most of the classes. Furthermore, the computing time spent in classification can be also decreased by removing some mislabeled training samples (see the final row of Table IV). Next, the performance of the SVM trained using 20 correct training samples is also given in Table IV. In other words, if the proposed method can detect and remove all mislabeled training samples, the performance of the proposed method should be the same as the SVM trained using the 20 correct training samples.).
Liu, Ge, Hao, and Tu are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, and Hao, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Tu to Liu before the effective filing date of the claimed invention in order to improve classification performance (cf. Tu, [Abstract, pg. 1573] Mislabeled training samples may have a negative effect on the performance of hyperspectral image classification. In order to solve this problem, a new density peak (DP) clustering-based noisy label detection method is proposed, which consists of the following steps. First, the distances among the training samples of each class are calculated using four representative distance metrics, i.e., the Euclidean distance (ED), orthogonal projection divergence (OPD), spectral information divergence (SID), and correlation coefficient (CC). Then, the local density of each training sample can be obtained using the DP clustering algorithm. Finally, a local density-based decision function is used to detect the noisy labels. The effectiveness of the proposed method is evaluated using the support vector machines on several real hyperspectral data sets. Experimental results demonstrate that the proposed noisy label detection method indeed helps in improving the classification performance.).
Regarding claim 2, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 1.
Hao teaches wherein using the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector includes: determining, by the processing resource, that the first angle is less than the second angle (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine the label value of an unlabeled vector black arrow by determining the first angle between W1 and the black arrow is less than the second angle between W2 and the black arrow and thus, black arrow is classified to class 1 (a).); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu teaches identifying, by the processing resource, the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle ([0074] At step 710, the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 3, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 2.
Liu teaches wherein using the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further includes: comparing, by the processing resource, the minimum angle with a threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a comparing, by the processing resource, the minimum angle with a threshold value pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 4, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 3.
Liu teaches wherein using the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further includes: identifying, by the processing resource, the vector as a high value labeling target where the minimum angle exceeds the threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0075] At step 714, the maximized distance of the vector product WF(x) is converted to a probability value indicating a likelihood that the input sample is in-distribution or OOD. For example, a tanh(.) operation is applied to transform the distance max(∥WF(x)∥) to a probability distribution of value between 0 and 1, and this probability indicates whether the given input x is in-domain or OOD. Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a where the minimum angle exceeds the threshold value pre-defined threshold distance, the input sample x may be determined to be OOD.; [0022] As the student model can be trained within a much shorter time than the BERT teacher model, the student model can be supplemented with out-of-distribution (OOD) training. Specifically, OOD samples can be generated from the given intent set, and the student model is identifying, by the processing resource, the unlabeled vector as a high value labeling target assigned with one more class label for an “OOD” class. The OOD samples are then fed to the student model together with inputs from the given intent dataset to train the student model for OOD identification.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 8, Liu teaches A system comprising: a processing resource; a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to ([0045] In some examples, memory 320 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 310) may cause the one or more processors to perform the methods described in further detail herein.):
receive a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector ([0027] An embodiment described herein provides that a receive a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector small dataset may be used for the neural network to learn characteristics of the radius of the input vector to the origin. In this way, an OOD vector may be identified when the OOD vector is sufficiently close to the origin (identified through learning), or when the OOD vector is orthogonal to all reference class vectors.; [0075] Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a pre-defined threshold distance, the input sample x may be determined to be OOD.);
Liu fails to teach identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target that is orthogonal to a space spanned by the plurality of labeled vectors by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector; create a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector; and train a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector.
Ge teaches calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector ([C. Similarity Measurement, pg. 32] For non-binary hypervectors, cosine similarity, defined by Eq. (1), is used to measure their similarity, focusing on the angle and ignoring the impact of the magnitude of hypervectors, where · denotes the magnitude. Unlike the inner product operation [12] of two vectors that affects magnitude and orientation, the cosine similarity only depends on the orientation. In most HD algorithms with non-binary hypervectors, cosine similarity is more often used than inner product. Moreover, when cos( , A B) is close to 1, this implies an extremely high level of similarity. For example, cos( , A B) = 1 indicates calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector two hypervectors A and B are identical. When they are at right angle, then cos( , A B) , = 0 and the two orthogonal vectors are considered dissimilar.; [(7) HD Computing for Semi-Supervised Learning, pg. 42] In [53], SemiHD has been proposed as a self-training or self-learning approach for semi-supervised learning, where the training data is composed of a small portion of labeled data and a large portion of unlabeled data. The SemiHD framework is depicted in Fig. 13 and the flow is illustrated as follows. 1). Encode all the data points, labeled and unlabeled, into HD space with d =10,000 dimensions. 2). Start training from the labeled data to generate k hypervectors, each representing one class. 3). Predict the label for unlabeled data points. Labeling is performed by checking the similarity of unlabeled data with all the class hypervectors, and return the label which shows the highest similarity.);
Liu and Ge are combinable for the same rationale as set forth above with respect to claim 1.
Hao teaches create a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine a labeling value of the unlabeled vector black arrow, resulting in the anchor create a newly labeled vector by classified into class 1 or class 2 dependent on the combination of said angles); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu, Ge, and Hao are combinable for the same rationale as set forth above with respect to claim 1.
Tu teaches identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target that is orthogonal to a space spanned by the plurality of labeled vectors by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including ([A. Calculating the Distances of the Training Samples, pg. 1575] Let x = {x1, x2,..., xM } refers to the original training set, in which M is the number of classes, and xm refers to the training samples in the mth class. For that is orthogonal to a space spanned by the plurality of labeled vectors two training samples belonging to the mth class, i.e., x j a and x j b, the distance dm ab between two samples can be measured. In this paper, four types of distances, i.e., the ED [44], orthogonal projection divergence (OPD) [45], spectral information divergence (SID) [46], and CC [20], are considered. The definitions of these distance metrics are presented in the following. 1) Euclidean Distance: dj ab = x j a − x j b 2 2. (6) by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including 2) Orthogonal Projection Divergence: dj ab = x j a T Wax j a + x j b T Wbx j b (7) where Wa = 1 − x j a(x j a T x j a)−1x j a T and Wb = 1 − x j b(x j b T x j b)−1x j b T . 3) Spectral Information Divergence: dj ab = u log u v +v log v u (8) where u = (x j a/ x j a) and v = (x j b/ x j b) refer to the desired probability vectors resulting from the pixel vectors x j a and x j b. 4) Correlation Coefficient: dj ab = covx j a, x j b varx j a · var x j b (9) where var(x j a) and var(x j b) refer to variances of the pixel vectors x j a and x j b.; [B. Calculating the Local Densities of the Training Samples, pg. 1576] First, the cutoff distance dm c is calculated as follows: dm c = Sm(t) s.t. t = Nm · (Nm − 1) 100 · p (11) where Sm is a matrix that sorts the nonzero elements in the upper triangular matrix of Dm from the smallest to the largest, p is a free parameter that will be analyzed in Section IV-B, and < · > refers to the round operation. With the above-obtained cutoff distance, the local densities ρm = {ρm 1 , ρm 2 ,..., ρm Nm } of the pixels in the mth class can be calculated as follows: ρm = e − Dm dm c 2 . (12); [C. Detecting the Mislabeled Samples, pg. 1576] Once the local densities of the training samples in different classes are obtained, the identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target mislabeled samples can be easily detected and removed as follows: Ym i = Xm i if ρm Nm ≥ λ · ρm ∅ Otherwise (13) where Y = {Y1, Y2,..., YM } refers to the resulting training set, in which the noisy labels have been detected and removed. λ is a free parameter.; Tu teaches using Euclidean Distance, Orthogonal Projection Divergence, Spectral Information Divergence, and Correlation Coefficient to calculate distances between sample vectors in classes and using said distances to determine density in sample data clusters to detect mislabeled samples (labeling target), removing noisy labels resulting in robust detection of mislabeled samples.)
train a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector ([IV. EXPERIMENTS, pg. 1576-1577] This paper adopts the SVM classifier to demonstrate the effectiveness of the based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector proposed noisy label detection method as it is one of the most widely used pixelwise classifiers. The SVM method is implemented using the LIBSVM library [47], and the parameters of the SVM are decided using a fivefold cross validation. Furthermore, in all experiments, three widely used quality indexes, i.e., the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient, are used to evaluate the performance of the proposed method. OA measures the percentage of all correctly classified pixels. AA represents the average value of the percentage of the correctly classified pixels for each class. The Kappa coefficient estimates the percentage of classified pixels corrected by the number of agreements that would be expected purely by chance. All experiments are repeated 10 times with randomly selected training samples so as to obtain the mean and standard variances of OA, AA and Kappa. The training sets are constructed using the samples in the ground truth. For each class, some pixels randomly selected from other classes will be added so as to simulate the “noisy label” problem.; [E. Performance Evaluation Using the SVM, pg. 1578-1579] It can be seen that the proposed method can train a machine-learning model improve the classification accuracies for most of the classes. Furthermore, the computing time spent in classification can be also decreased by removing some mislabeled training samples (see the final row of Table IV). Next, the performance of the SVM trained using 20 correct training samples is also given in Table IV. In other words, if the proposed method can detect and remove all mislabeled training samples, the performance of the proposed method should be the same as the SVM trained using the 20 correct training samples.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 9, Liu, as modified by Ge, Hao, and Tu, teaches The system of claim 8.
Hao teaches wherein the instructions that when executed by the processing resource cause the processing resource to use the combination of the first angle and the second angle to determine the labeling value of the unlabeled vector include instructions that cause the processing resource to: determine that the first angle is less than the second angle (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine the label value of an unlabeled vector black arrow by determining the first angle between W1 and the black arrow is less than the second angle between W2 and the black arrow and thus, black arrow is classified to class 1 (a).); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu teaches identify the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle ([0074] At step 710, the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 10, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 9.
Liu teaches wherein the instructions that when executed by the processing resource cause the processing resource to the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further include instructions that cause the processing resource to: compare the minimum angle with a threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a compare the minimum angle with a threshold value pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 11, Liu, as modified by Ge, Hao, and Tu, teaches The system of claim 10.
Liu teaches wherein the instructions that when executed by the processing resource cause the processing resource to the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further include instructions that cause the processing resource to: identify the unlabeled vector as a high value labeling target where the minimum angle exceeds the threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0075] At step 714, the maximized distance of the vector product WF(x) is converted to a probability value indicating a likelihood that the input sample is in-distribution or OOD. For example, a tanh(.) operation is applied to transform the distance max(∥WF(x)∥) to a probability distribution of value between 0 and 1, and this probability indicates whether the given input x is in-domain or OOD. Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a where the minimum angle exceeds the threshold value pre-defined threshold distance, the input sample x may be determined to be OOD.; [0022] As the student model can be trained within a much shorter time than the BERT teacher model, the student model can be supplemented with out-of-distribution (OOD) training. Specifically, OOD samples can be generated from the given intent set, and the student model is identify the unlabeled vector as a high value labeling target assigned with one more class label for an “OOD” class. The OOD samples are then fed to the student model together with inputs from the given intent dataset to train the student model for OOD identification.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Claims 5, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Ge, Hao, Tu, and further in view of Efstathiou et al. (U.S. Pre-Grant Publication No. 2019/0370219, hereinafter ‘Efstathiou').
Regarding claim 5, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 3.
Liu, as modified by Ge, Hao, and Tu, fails to teach wherein the threshold value is user programmable.
Efstathiou teaches wherein the threshold value is user programmable ([0115] The AlSynth method can be used as a stand-alone method or as a task for one or more agents within the multi-RL methods described herein. As a stand-alone method, it requires the threshold value is user programmable user to define a particular threshold value from the set: (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). This value represents the classifier's confidence score. The classifier, when classifying a particular instance, will output a confidence score according to the confidence that the instance belongs to a particular class (ranging from 0.0 representing 0% confidence, to 1.0 representing 100% confidence). Each of the instances that has a confidence score that is greater than or equal to the threshold is labeled according to the result of the classification and added to a set of labeled data. This set of labeled data may be added into the original labeled train set to provide additional data to help train the system.).
Liu, Ge, Hao, Tu, and Efstathiou are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, Hao, and Tu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Efstathiou to Liu before the effective filing date of the claimed invention in order to output a confidence score according to the confidence that the instance belongs to a particular class (cf. Efstathiou, [0115] The AlSynth method can be used as a stand-alone method or as a task for one or more agents within the multi-RL methods described herein. As a stand-alone method, it requires the user to define a particular threshold value from the set: (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). This value represents the classifier's confidence score. The classifier, when classifying a particular instance, will output a confidence score according to the confidence that the instance belongs to a particular class (ranging from 0.0 representing 0% confidence, to 1.0 representing 100% confidence). Each of the instances that has a confidence score that is greater than or equal to the threshold is labeled according to the result of the classification and added to a set of labeled data. This set of labeled data may be added into the original labeled train set to provide additional data to help train the system.).
Regarding claim 12, Liu, as modified by Ge, Hao, and Tu, teaches The system of claim 10.
Liu, as modified by Ge, Hao, and Tu, fails to teach wherein the threshold value is user programmable.
Efstathiou teaches wherein the threshold value is user programmable ([0115] The AlSynth method can be used as a stand-alone method or as a task for one or more agents within the multi-RL methods described herein. As a stand-alone method, it requires the threshold value is user programmable user to define a particular threshold value from the set: (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). This value represents the classifier's confidence score. The classifier, when classifying a particular instance, will output a confidence score according to the confidence that the instance belongs to a particular class (ranging from 0.0 representing 0% confidence, to 1.0 representing 100% confidence). Each of the instances that has a confidence score that is greater than or equal to the threshold is labeled according to the result of the classification and added to a set of labeled data. This set of labeled data may be added into the original labeled train set to provide additional data to help train the system.).
Liu, Ge, Hao, Tu, and Efstathiou are combinable for the same rationale as set forth above with respect to claim 5.
Claims 6-7, 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Ge, Hao, Tu, and further in view of Tuia et al. (NPL: "A survey of active learning algorithms for supervised remote sensing image classification", hereinafter ‘Tuia').
Regarding claim 6, Liu, as modified by Ge, Hao, and Tu, teaches The method of claim 1.
Liu, as modified by Ge, Hao, and Tu, fails to teach the method further comprising: using, by the processing resource, the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors.
Tuia teaches the method further comprising: using, by the processing resource, the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] An active learning process requires interaction between the user and the model: the first provides the labeled information and the knowledge about the desired classes, while the latter provides both its own interpretation of the distribution of the classes and the most relevant pixels that are needed in order to solve the discrepancies encountered. This point is crucial for the success of an to rank the unlabeled vector relative to other unlabeled vectors active learning algorithm: the machine needs a strategy to rank the pixels in the pool. These using, by the processing resource, the labeling value of the unlabeled vector along with the result of at least one other heuristic strategies, or heuristics, differentiate the algorithms proposed in the next sections and can be divided into three main families [21]. 1) Committee-based heuristics (Section III). 2) Large margin-based heuristics (Section IV). 3) Posterior probability-based heuristics (Section V). A last family of active learning heuristics, the cluster-based, has recently being proposed in the community [22]: clusterbased heuristics aim at pruning a hierarchical clustering tree until the resulting clusters are consistent with the labels provided by the user. Therefore, these strategies rely on an unsupervised model, rather than on a predictive model. Since the aim of these heuristics is different form that of the other families presented, they will not be detailed in this survey.).
Liu, Ge, Hao, Tu, and Tuia are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, Hao, and Tu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Tuia to Liu before the effective filing date of the claimed invention in order to strategize active learning while classifying unlabeled data (cf. Tuia, [II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] An active learning process requires interaction between the user and the model: the first provides the labeled information and the knowledge about the desired classes, while the latter provides both its own interpretation of the distribution of the classes and the most relevant pixels that are needed in order to solve the discrepancies encountered. This point is crucial for the success of an active learning algorithm: the machine needs a strategy to rank the pixels in the pool . These strategies, or heuristics, differentiate the algorithms proposed in the next sections and can be divided into three main families [21]. 1) Committee-based heuristics (Section III). 2) Large margin-based heuristics (Section IV). 3) Posterior probability-based heuristics (Section V). A last family of active learning heuristics, the cluster-based, has recently being proposed in the community [22]: clusterbased heuristics aim at pruning a hierarchical clustering tree until the resulting clusters are consistent with the labels provided by the user. Therefore, these strategies rely on an unsupervised model, rather than on a predictive model. Since the aim of these heuristics is different form that of the other families presented, they will not be detailed in this survey.).
Regarding claim 7, Liu, as modified by Ge, Hao, Tu, and Tuia, teaches The method of claim 6.
Tuia teaches wherein the at least one other heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] [A. Shannon's entropy heuristic Normalized Entropy Query-by-Bagging (nEQB), pg. 607] In the implementations of [25], bagging [29] is proposed to build the committee: first, training sets built on a draw with replacement of the original data are defined. These draws account for a part of the available labeled pixels only. Then, each set is used to train a classifier and to predict the labels of the candidates. At the end of the procedure, labelings are provided for each candidate pixel. In [21], the entropy of the distribution of the predictions is used as heuristic.; [IV. LARGE-MARGIN-BASED ACTIVE LEARNING, pg. 608] The second family of methods is specific to margin-based classifiers. Methods such as SVM are naturally good base methods for active learning: the distance from decision hyperplane heuristic distance to the separating hyperplane, that is the absolute value of the decision function without the sign operator, is a straightforward way of estimating the classifier confidence on an unseen sample.).
Liu, Ge, Hao, Tu, and Tuia are combinable for the same rationale as set forth above with respect to claim 6.
Regarding claim 13, Liu, as modified by Ge, Hao, and Tu, teaches The system of claim 8.
Liu, as modified by Ge, Hao, and Tu, fails to teach wherein the instructions that when executed by the processing resource further cause the processing resource to: using the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors.
Tuia teaches wherein the instructions that when executed by the processing resource further cause the processing resource to: using the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] An active learning process requires interaction between the user and the model: the first provides the labeled information and the knowledge about the desired classes, while the latter provides both its own interpretation of the distribution of the classes and the most relevant pixels that are needed in order to solve the discrepancies encountered. This point is crucial for the success of an to rank the unlabeled vector relative to other unlabeled vectors active learning algorithm: the machine needs a strategy to rank the pixels in the pool. These using the labeling value of the unlabeled vector along with the result of at least one other heuristic strategies, or heuristics, differentiate the algorithms proposed in the next sections and can be divided into three main families [21]. 1) Committee-based heuristics (Section III). 2) Large margin-based heuristics (Section IV). 3) Posterior probability-based heuristics (Section V). A last family of active learning heuristics, the cluster-based, has recently being proposed in the community [22]: clusterbased heuristics aim at pruning a hierarchical clustering tree until the resulting clusters are consistent with the labels provided by the user. Therefore, these strategies rely on an unsupervised model, rather than on a predictive model. Since the aim of these heuristics is different form that of the other families presented, they will not be detailed in this survey.).
Liu, Ge, Hao, Tu, and Tuia are combinable for the same rationale as set forth above with respect to claim 6.
Regarding claim 14, Liu, as modified by Ge, Hao, Tu, and Tuia, teaches The system of claim 13.
Tuia teaches wherein the at least one other heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] [A. Shannon's entropy heuristic Normalized Entropy Query-by-Bagging (nEQB), pg. 607] In the implementations of [25], bagging [29] is proposed to build the committee: first, training sets built on a draw with replacement of the original data are defined. These draws account for a part of the available labeled pixels only. Then, each set is used to train a classifier and to predict the labels of the candidates. At the end of the procedure, labelings are provided for each candidate pixel. In [21], the entropy of the distribution of the predictions is used as heuristic.; [IV. LARGE-MARGIN-BASED ACTIVE LEARNING, pg. 608] The second family of methods is specific to margin-based classifiers. Methods such as SVM are naturally good base methods for active learning: the distance from decision hyperplane heuristic distance to the separating hyperplane, that is the absolute value of the decision function without the sign operator, is a straightforward way of estimating the classifier confidence on an unseen sample.).
Liu, Ge, Hao, Tu, and Tuia are combinable for the same rationale as set forth above with respect to claim 6.
Claims 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Ge, Hao, Tu, and further in view of Arnas et al. (NPL: "The n-dimensional k-vector and its application to orthogonal range searching", hereinafter ‘Arnas').
Regarding claim 15, Liu teaches A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a computer system, causes the one or more processing resources to ([0045] In some examples, memory 320 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 310) may cause the one or more processors to perform the methods described in further detail herein.):
receive a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector ([0027] An embodiment described herein provides that a receive a set of vectors including one or more unlabeled vectors and a plurality of labeled vectors including at least a first labeled vector, and a second labeled vector small dataset may be used for the neural network to learn characteristics of the radius of the input vector to the origin. In this way, an OOD vector may be identified when the OOD vector is sufficiently close to the origin (identified through learning), or when the OOD vector is orthogonal to all reference class vectors.; [0075] Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a pre-defined threshold distance, the input sample x may be determined to be OOD.);
Liu fails to teach search for examples that are orthogonal to a problem space spanned by the plurality of labeled vectors that has not vet been explored to identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector; create a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector; and train a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector.
Ge teaches calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector ([C. Similarity Measurement, pg. 32] For non-binary hypervectors, cosine similarity, defined by Eq. (1), is used to measure their similarity, focusing on the angle and ignoring the impact of the magnitude of hypervectors, where · denotes the magnitude. Unlike the inner product operation [12] of two vectors that affects magnitude and orientation, the cosine similarity only depends on the orientation. In most HD algorithms with non-binary hypervectors, cosine similarity is more often used than inner product. Moreover, when cos( , A B) is close to 1, this implies an extremely high level of similarity. For example, cos( , A B) = 1 indicates calculating a first angle between the unlabeled vector and the first labeled vector, and a second angle between the unlabeled vector and the second labeled vector two hypervectors A and B are identical. When they are at right angle, then cos( , A B) , = 0 and the two orthogonal vectors are considered dissimilar.; [(7) HD Computing for Semi-Supervised Learning, pg. 42] In [53], SemiHD has been proposed as a self-training or self-learning approach for semi-supervised learning, where the training data is composed of a small portion of labeled data and a large portion of unlabeled data. The SemiHD framework is depicted in Fig. 13 and the flow is illustrated as follows. 1). Encode all the data points, labeled and unlabeled, into HD space with d =10,000 dimensions. 2). Start training from the labeled data to generate k hypervectors, each representing one class. 3). Predict the label for unlabeled data points. Labeling is performed by checking the similarity of unlabeled data with all the class hypervectors, and return the label which shows the highest similarity.);
Liu and Ge are combinable for the same rationale as set forth above with respect to claim 1.
Hao teaches create a newly labeled vector by using a combination of the first angle and the second angle to determine a labeling value of the unlabeled vector (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine a labeling value of the unlabeled vector black arrow, resulting in the anchor create a newly labeled vector by classified into class 1 or class 2 dependent on the combination of said angles); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu, Ge, and Hao are combinable for the same rationale as set forth above with respect to claim 1.
Tu teaches identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including ([A. Calculating the Distances of the Training Samples, pg. 1575] Let x = {x1, x2,..., xM } refers to the original training set, in which M is the number of classes, and xm refers to the training samples in the mth class. For two training samples belonging to the mth class, i.e., x j a and x j b, the distance dm ab between two samples can be measured. In this paper, four types of distances, i.e., the ED [44], orthogonal projection divergence (OPD) [45], spectral information divergence (SID) [46], and CC [20], are considered. The definitions of these distance metrics are presented in the following. 1) Euclidean Distance: dj ab = x j a − x j b 2 2. (6) by applying an orthogonality heuristic to each labeled vector of the plurality of labeled vectors including 2) Orthogonal Projection Divergence: dj ab = x j a T Wax j a + x j b T Wbx j b (7) where Wa = 1 − x j a(x j a T x j a)−1x j a T and Wb = 1 − x j b(x j b T x j b)−1x j b T . 3) Spectral Information Divergence: dj ab = u log u v +v log v u (8) where u = (x j a/ x j a) and v = (x j b/ x j b) refer to the desired probability vectors resulting from the pixel vectors x j a and x j b. 4) Correlation Coefficient: dj ab = covx j a, x j b varx j a · var x j b (9) where var(x j a) and var(x j b) refer to variances of the pixel vectors x j a and x j b.; [B. Calculating the Local Densities of the Training Samples, pg. 1576] First, the cutoff distance dm c is calculated as follows: dm c = Sm(t) s.t. t = Nm · (Nm − 1) 100 · p (11) where Sm is a matrix that sorts the nonzero elements in the upper triangular matrix of Dm from the smallest to the largest, p is a free parameter that will be analyzed in Section IV-B, and < · > refers to the round operation. With the above-obtained cutoff distance, the local densities ρm = {ρm 1 , ρm 2 ,..., ρm Nm } of the pixels in the mth class can be calculated as follows: ρm = e − Dm dm c 2 . (12); [C. Detecting the Mislabeled Samples, pg. 1576] Once the local densities of the training samples in different classes are obtained, the identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target mislabeled samples can be easily detected and removed as follows: Ym i = Xm i if ρm Nm ≥ λ · ρm ∅ Otherwise (13) where Y = {Y1, Y2,..., YM } refers to the resulting training set, in which the noisy labels have been detected and removed. λ is a free parameter.; Tu teaches using Euclidean Distance, Orthogonal Projection Divergence, Spectral Information Divergence, and Correlation Coefficient to calculate distances between sample vectors in classes and using said distances to determine density in sample data clusters to detect mislabeled samples (labeling target), removing noisy labels resulting in robust detection of mislabeled samples.)
train a machine-learning model based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector ([IV. EXPERIMENTS, pg. 1576-1577] This paper adopts the SVM classifier to demonstrate the effectiveness of the based on a set of labeled vectors including the plurality of labeled vectors and the newly labeled vector proposed noisy label detection method as it is one of the most widely used pixelwise classifiers. The SVM method is implemented using the LIBSVM library [47], and the parameters of the SVM are decided using a fivefold cross validation. Furthermore, in all experiments, three widely used quality indexes, i.e., the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient, are used to evaluate the performance of the proposed method. OA measures the percentage of all correctly classified pixels. AA represents the average value of the percentage of the correctly classified pixels for each class. The Kappa coefficient estimates the percentage of classified pixels corrected by the number of agreements that would be expected purely by chance. All experiments are repeated 10 times with randomly selected training samples so as to obtain the mean and standard variances of OA, AA and Kappa. The training sets are constructed using the samples in the ground truth. For each class, some pixels randomly selected from other classes will be added so as to simulate the “noisy label” problem.; [E. Performance Evaluation Using the SVM, pg. 1578-1579] It can be seen that the proposed method can train a machine-learning model improve the classification accuracies for most of the classes. Furthermore, the computing time spent in classification can be also decreased by removing some mislabeled training samples (see the final row of Table IV). Next, the performance of the SVM trained using 20 correct training samples is also given in Table IV. In other words, if the proposed method can detect and remove all mislabeled training samples, the performance of the proposed method should be the same as the SVM trained using the 20 correct training samples.).
Liu, Ge, Hao, and Tu are combinable for the same rationale as set forth above with respect to claim 1.
Arnas teaches search for examples that are orthogonal to a problem space spanned by the plurality of labeled vectors that has not vet been explored to identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target ([2. General overview of the algorithm, pg. 2] The n-dimensional k-vector is a numerical algorithm specifically devised to search for examples that are orthogonal to a problem space perform orthogonal range searches in multidimensional static databases. The algorithm first finds the most convenient dimension in which to perform a projection of the database by estimating the number of expected retrieved elements in each dimension. This is done using three auxiliary databases that first estimate the number of elements in the searching range in each dimension, and then, identify which points are contained in the projection. Since up to this point the elements identified have only been checked in one dimension, a brute force approach follows to assess the remaining dimensions. Therefore, only a fraction of the database is checked during the searching process. As such, the n-dimensional k-vector can be regarded as a modified projection method in one dimension, where an auxiliary database is used rather than a binary search. However, an additional idea is introduced in the algorithm. Instead of searching in the whole database at the same time, the n-dimensional k-vector performs multiple to identify an unlabeled vector of the one or more unlabeled vectors as an ignored labeling target searches in subsets of the database, which are defined in such a way that they spanned by the plurality of labeled vectors that has not vet been explored only contain elements in a known range from a given dimension selected during the preprocessing. This allows the n-dimensional k-vector to asses two dimensions at the same time: the first one due to the structure of the database, and the second one due to the projection method.; [2. General overview of the algorithm, pg. 3-4] Once the preprocessing is finished, the n-dimensional k-vector is able to perform the orthogonal search. This is done by following these steps, which are also presented schematically in Fig. 3)
Liu, Ge, Hao, Tu, and Arnas are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, Hao, and Tu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Arnas to Liu before the effective filing date of the claimed invention in order to use the k-vector, a range searching technique for one dimension that identifies the number of elements contained in the searching range and predict and select the best approach to deal with each dimension (cf. Arnas, [Abstract, pg. 1] This work focuses on the definition and study of the n-dimensional k-vector, an algorithm devised to perform orthogonal range searching in static databases with multiple dimensions. The methodology first finds the order in which to search the dimensions, and then, performs the search using a modified projection method. In order to determine the dimension order, the algorithm uses the k-vector, a range searching technique for one dimension that identifies the number of elements contained in the searching range. Then, using this information, the algorithm predicts and selects the best approach to deal with each dimension. The algorithm has a worst case complexity of O(nd(k/n)2/d), where k is the number of elements retrieved, n is the number of elements in the database, and d is the number of dimensions of the database. This work includes a detailed description of the methodology as well as a study of the algorithm performance.).
Regarding claim 16, Liu, as modified by Ge, Hao, Tu, and Arnas, teaches The non-transitory computer-readable storage medium of claim 15.
Hao teaches wherein the instructions that when executed by the one or more processing resources of the computer system cause the one or more processing resources to use the combination of the first angle and the second angle to determine the labeling value of the unlabeled vector include instructions that cause the one or more processing resources to:
determine that the first angle is less than the second angle (As shown in the figure below, Hao teaches using a combination of the first angle W1 and second angle W2 to determine the label value of an unlabeled vector black arrow by determining the first angle between W1 and the black arrow is less than the second angle between W2 and the black arrow and thus, black arrow is classified to class 1 (a).); and
PNG
media_image1.png
293
418
media_image1.png
Greyscale
Liu teaches identify the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle ([0074] At step 710, the first angle as a minimum angle based at least in part on determining that the first angle is less than the second angle minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, Tu, and Arnas are combinable for the same rationale as set forth above with respect to claim 15.
Regarding claim 17, Liu, as modified by Ge, Hao, Tu, and Arnas, teaches The non-transitory computer-readable storage medium of claim 16.
Liu teaches wherein the instructions that when executed by the one or more processing resources of the computer system cause the one or more processing resources to the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further include instructions that cause the one or more processing resources to:
compare the minimum angle with a threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0061] At step 410, the neural network determines whether the input sample is in-distribution or out-of-distribution based on the generated classification output. For example, a compare the minimum angle with a threshold value pre-defined threshold may be used to determine whether the input sample is in-distribution or OOD depending on whether a classification probability is greater or less than the pre-defined threshold.).
Liu, Ge, Hao, Tu, and Arnas are combinable for the same rationale as set forth above with respect to claim 15.
Regarding claim 18, Liu, as modified by Ge, Hao, Tu, and Arnas, teaches The non-transitory computer-readable storage medium of claim 17.
Liu teaches wherein the instructions that when executed by the one or more processing resources of the computer system cause the one or more processing resources to the combination of the first angle and the second angle to determine a labeling value of the unlabeled vector further include instructions that cause the one or more processing resources to: identify the unlabeled vector as a high value labeling target where the minimum angle exceeds the threshold value ([0074] At step 710, the minimum angle made by the vector representation (F(x)) of an input x against an in-domain class representation that is the closest to the vector F(x) is computed by maximizing a distance of the vector product WF(x), e.g., max(∥WF(x)∥).; [0075] At step 714, the maximized distance of the vector product WF(x) is converted to a probability value indicating a likelihood that the input sample is in-distribution or OOD. For example, a tanh(.) operation is applied to transform the distance max(∥WF(x)∥) to a probability distribution of value between 0 and 1, and this probability indicates whether the given input x is in-domain or OOD. Based on the probability, it can be determined whether the input sample is in-distribution or OOD. In particular, when the vector representation of the input sample F(x) is orthogonal to the number of reference class vectors or is close to the origin for less than a where the minimum angle exceeds the threshold value pre-defined threshold distance, the input sample x may be determined to be OOD.; [0022] As the student model can be trained within a much shorter time than the BERT teacher model, the student model can be supplemented with out-of-distribution (OOD) training. Specifically, OOD samples can be generated from the given intent set, and the student model is identify the unlabeled vector as a high value labeling target assigned with one more class label for an “OOD” class. The OOD samples are then fed to the student model together with inputs from the given intent dataset to train the student model for OOD identification.).
Liu, Ge, Hao, Tu, and Arnas are combinable for the same rationale as set forth above with respect to claim 15.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Ge, Hao, Tu, Arnas, and further in view of Efstathiou.
Regarding claim 19, Liu, as modified by Ge, Hao, Tu, and Arnas, teaches The non-transitory computer-readable storage medium of claim 17.
Liu, as modified by Ge, Hao, Tu, and Arnas, fails to teach wherein the threshold value is user programmable.
Efstathiou teaches wherein the threshold value is user programmable ([0115] The AlSynth method can be used as a stand-alone method or as a task for one or more agents within the multi-RL methods described herein. As a stand-alone method, it requires the threshold value is user programmable user to define a particular threshold value from the set: (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). This value represents the classifier's confidence score. The classifier, when classifying a particular instance, will output a confidence score according to the confidence that the instance belongs to a particular class (ranging from 0.0 representing 0% confidence, to 1.0 representing 100% confidence). Each of the instances that has a confidence score that is greater than or equal to the threshold is labeled according to the result of the classification and added to a set of labeled data. This set of labeled data may be added into the original labeled train set to provide additional data to help train the system.).
Liu, Ge, Hao, Tu, Arnas, and Efstathiou are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, Hao, Tu, and Arnas, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Efstathiou to Liu before the effective filing date of the claimed invention in order to output a confidence score according to the confidence that the instance belongs to a particular class (cf. Efstathiou, [0115] The AlSynth method can be used as a stand-alone method or as a task for one or more agents within the multi-RL methods described herein. As a stand-alone method, it requires the user to define a particular threshold value from the set: (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). This value represents the classifier's confidence score. The classifier, when classifying a particular instance, will output a confidence score according to the confidence that the instance belongs to a particular class (ranging from 0.0 representing 0% confidence, to 1.0 representing 100% confidence). Each of the instances that has a confidence score that is greater than or equal to the threshold is labeled according to the result of the classification and added to a set of labeled data. This set of labeled data may be added into the original labeled train set to provide additional data to help train the system.).
Claims 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Ge, Hao, Tu, Arnas, and further in view of Tuia.
Regarding claim 20, Liu, as modified by Ge, Hao, Tu, and Arnas, teaches The non-transitory computer-readable storage medium of claim 15.
Liu, as modified by Ge, Hao, Tu, and Arnas, fails to teach wherein the instructions that when executed by the one or more processing resources of the computer system further cause the one or more processing resources to: using the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors.
Tuia teaches wherein the instructions that when executed by the one or more processing resources of the computer system further cause the one or more processing resources to:
using the labeling value of the unlabeled vector along with the result of at least one other heuristic to rank the unlabeled vector relative to other unlabeled vectors ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] An active learning process requires interaction between the user and the model: the first provides the labeled information and the knowledge about the desired classes, while the latter provides both its own interpretation of the distribution of the classes and the most relevant pixels that are needed in order to solve the discrepancies encountered. This point is crucial for the success of an to rank the unlabeled vector relative to other unlabeled vectors active learning algorithm: the machine needs a strategy to rank the pixels in the pool. These using the labeling value of the unlabeled vector along with the result of at least one other heuristic strategies, or heuristics, differentiate the algorithms proposed in the next sections and can be divided into three main families [21]. 1) Committee-based heuristics (Section III). 2) Large margin-based heuristics (Section IV). 3) Posterior probability-based heuristics (Section V). A last family of active learning heuristics, the cluster-based, has recently being proposed in the community [22]: clusterbased heuristics aim at pruning a hierarchical clustering tree until the resulting clusters are consistent with the labels provided by the user. Therefore, these strategies rely on an unsupervised model, rather than on a predictive model. Since the aim of these heuristics is different form that of the other families presented, they will not be detailed in this survey.).
Liu, Ge, Hao, Tu, Arnas, and Tuia are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Liu, Ge, Hao, Tu, and Arnas, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Tuia to Liu before the effective filing date of the claimed invention in order to strategize active learning while classifying unlabeled data (cf. Tuia, [II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] An active learning process requires interaction between the user and the model: the first provides the labeled information and the knowledge about the desired classes, while the latter provides both its own interpretation of the distribution of the classes and the most relevant pixels that are needed in order to solve the discrepancies encountered. This point is crucial for the success of an active learning algorithm: the machine needs a strategy to rank the pixels in the pool . These strategies, or heuristics, differentiate the algorithms proposed in the next sections and can be divided into three main families [21]. 1) Committee-based heuristics (Section III). 2) Large margin-based heuristics (Section IV). 3) Posterior probability-based heuristics (Section V). A last family of active learning heuristics, the cluster-based, has recently being proposed in the community [22]: clusterbased heuristics aim at pruning a hierarchical clustering tree until the resulting clusters are consistent with the labels provided by the user. Therefore, these strategies rely on an unsupervised model, rather than on a predictive model. Since the aim of these heuristics is different form that of the other families presented, they will not be detailed in this survey.).
Regarding claim 21, Liu, as modified by Ge, Hao, Tu, Arnas, and Tuia, teaches The non-transitory computer-readable storage medium of claim 20.
Tuia teaches wherein the at least one other heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic ([II. ACTIVE LEARNING: CONCEPTS AND DEFINITIONS, pg. 607] [A. Shannon's entropy heuristic Normalized Entropy Query-by-Bagging (nEQB), pg. 607] In the implementations of [25], bagging [29] is proposed to build the committee: first, training sets built on a draw with replacement of the original data are defined. These draws account for a part of the available labeled pixels only. Then, each set is used to train a classifier and to predict the labels of the candidates. At the end of the procedure, labelings are provided for each candidate pixel. In [21], the entropy of the distribution of the predictions is used as heuristic.; [IV. LARGE-MARGIN-BASED ACTIVE LEARNING, pg. 608] The second family of methods is specific to margin-based classifiers. Methods such as SVM are naturally good base methods for active learning: the distance from decision hyperplane heuristic distance to the separating hyperplane, that is the absolute value of the decision function without the sign operator, is a straightforward way of estimating the classifier confidence on an unseen sample.).
Liu, Ge, Hao, Tu, Arnas, and Tuia are combinable for the same rationale as set forth above with respect to claim 20.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAGGIE MAIDO whose telephone number is (703) 756-1953. The examiner can normally be reached M-Th: 6am - 4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MM/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights