DETAILED ACTION
Status of Claims
Claim(s) 1, 3-6, 8-12, 14-17 and 19-20 are pending and are examined herein.
Claim(s) 1, 3, 11, 12, and 14 have been Amended. Claims 2 and 13 previously Cancelled. Claims 7 and 18 are now Cancelled.
Claim(s) 1, 3-6, 8-12, 14-17 and 19-20 rejected under 35 U.S.C. § 101 and 35 U.S.C. § 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on August 22, 2025 has been entered. Claims 1, 3-6, 8-12, 14-17 and 19-20 are pending in the application. Applicant’s amendments to claims have overcome the objections and rejection under 35 U.S.C. § 112(b) previously set forth in the Non-Final Office Action mailed on May 22, 2025. Applicant’s amendments to the claims have been fully considered and are addressed in the rejections below.
Response to Arguments
Applicant's arguments, with respect to the rejection under 35 U.S.C. § 101 filed on 08/22/2025, have been fully considered but they are not persuasive.
Applicant’s argument (Pp. 16-17 of the remarks): Applicant submits that even if claim 1 were directed to such a concept, the alleged abstract idea does not fall within any of the subject matter groupings of abstract ideas enumerated in the 2019 Guidance (i.e. "Mathematical concepts," "Certain methods of organizing human activity," or "Mental Processes"). Therefore, for at least this reason, claim 1 does not recite an abstract idea, failing Prong One of the USPTO's required analysis.
Examiner's response: The examiner respectfully disagrees with the applicant assertion that the claims do not recite a judicial exception under Step 2A, Prong One of the subject matter eligibility analysis. As explained in the Office Action, the claims of the present application are primarily directed to assessing dataset quality using multiple attributes and providing recommendations and explanations regarding the assessment. The assessment involves scoring data with respect to each attribute. This include assessing boundary complexity, identifying overlapping class regions, identifying noisy and confusing labels, and performing typical statistical data cleaning and preprocessing steps to prepare the dataset. These steps collectively constitute statistical data analysis and evaluation that can be performed in the human mind with physical aid (e.g., pen and paper or slide rule). Thus, they fall under the category of mental processes—concepts performed in the human mind (including an observation, evaluation, judgment, and opinion) (see MPEP § 2106.04(a)(2), subsection III). For example, a data scientist can manually perform such data analysis using different mathematical/statistical measures to assess and prepare a dataset before using it for training a machine learning model.
Applicant’s argument (Pp. 17-18 of the remarks): Applicant submits that the amended claims integrate any alleged abstract ideas into practical applications, and that therefore the claims are not directed to any abstract ideas. By integrating these features into a machine-learning environment, claim 1 integrates the features and any associated abstract ideas - into a practical application (e.g. implementing a user-updated recommendation), thereby rendering claim 1 subject matter eligible. Applicant respectfully submits that claim 1 integrates its features into a practical application of assessing a quality of a dataset at least by providing improvements to existing datasets used in building machine-learning models. For example, paragraph [0019] of the as-filed specification states (emphasis added).
Examiner's response: The examiner respectfully disagrees. The amended limitations, including implementing user-updated recommendations, remain directed to the abstract idea of mental processes. The recited steps such as removing or correcting noisy and confusing data labels, performing probabilistic analysis to estimate class confidence, comparing confusing data points, aggregating noisy/confusing data points, adjusting feature weights, and applying data transformation, are all forms of statistical data evaluation and preparation that can be performed manually or with physical aid (e.g., pen and paper). Additionally, merely implementing the user-updated recommendation on a computer cannot preclude the claimed steps from being practically performed in the human mind. This merely describes applying user input or feedback using a computer component. The preceding steps represent statistical data cleaning and preprocessing operations that can be manually derived by an individual.
Furthermore, providing recommendations and explanations regarding data quality is itself a mental process. For example, a machine learning developer or data scientist can perform such data analysis using different data assessment quality attributes and then provide recommendation and explanation of the findings. The recitation of “thereby improving an ability of the machine-learning model to accurately classify the one or more data points” merely describes an intended result or field of use limitation, not an active step in the claimed method. See MPEP § 2111.01.
Accordingly, the examiner submits that the claims are directed to a judicial exception (i.e., data analysis and preparation). The identified additional elements, whether considered individually or in combination with the judicial exception, are not sufficient to integrate the judicial exception into a practical application or amount to significantly more.
In view of the above, the applicant’s arguments are not persuasive, and the rejection under 35 U.S.C. § 101 is maintained.
Applicant's arguments, with respect to the rejection under 35 U.S.C. § 103 filed on 08/22/2025 (see remarks Pp. 20-21) have been fully considered but are moot in view of the new grounds of rejection necessitated by amendments.
The examiner refers to the updated rejection under 35 U.S.C. § 103 for more details.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim(s) 1, 3-6, 8-12, 14-17 and 19-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding Amended Claim 1:
The claim recites the limitation: “correcting identified noisy and confusing data labels by clustering the identified noisy and confusing data labels based upon the classes defined within the dataset;” lines 23-25. This limitation introduces new matter not originally disclosed. Specifically, the specification ([0030]-[0038]) describes clustering being applied to all data points to identify noisy/confusing points, not as a step that acts on the already-identified noisy/confusing points. The claim, however, suggests that clustering is applied directly to the noisy/confusing points. The specification describes that clustering is applied to data points first, noisy/confusing points are identified, and then recommendation or corrections for noisy and confusing points is provided. Accordingly, the claim introduces a new matter that is different from what is disclosed in the specification. The specification does not provide clear written description support for the amended feature. For examination purposes, the examiner interprets this limitation as: clustering data points based upon classes defined within the dataset, identifying noisy/confusing data points, and correcting data labels.
The claim recites the limitation: “comparing confusing data points identified by the machine learning model and confusing data points identified through clustering;” lines 27-28. Paragraphs [0030]-[0031] of the specification describe identifying confusing points using an algorithm that calculates label purity, performs class-wise sample probability analysis, and generate a confidence matrix. The specification further describes comparing these confusing points found by the algorithm and the confusing point regions identified through the clustering. However, the specification does not disclose identifying confusing points using a machine learning model. Therefore, the comparison limitation introduces subject matter that was not originally disclosed. For examination purposes, the examiner interprets this limitation as comparing confusing/noisy points identified by an algorithm and confusing/noisy points identified through the clustering.
The claim recites the limitation: “adjusting feature weights to reduce unnecessary boundary complexity;” lines 30-31. Paragraphs [0025]-[0037] describes an optimization framework that weights features to assess boundary complexity, where features that contribute unnecessarily are weighted lower and features that help minimize the overall error rate of the machine-learning model will be weighted higher. The specification primarily uses this framework to provide a boundary complexity score assessment, not to actively modify feature weights in the dataset as a corrective step after the assessment. Thus, the specification fails to provide written description support for actually adjusting feature weights or an example where the method implements feature weights adjustment in order to reduce boundary complexity in the dataset. Accordingly, the claim language “adjusting feature weights to reduce unnecessary boundary complexity” goes beyond what the specification describes. For examination, the examiner interprets this limitation as weighting features based on their contribution to boundary complexity.
Regarding Amended Independent Claims 11 and 12:
The claims recite similar limitation as corresponding claim 1 and are rejected for similar reasons as claim 1 using similar rationale.
Regarding claims 3-6, 8-10, 14-17 and 19-20, dependent claims inherit the deficiencies of the respective parent claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself. Applicant is advised to consult MPEP 2106 for more details of the analysis.
Under Step 1 analysis,
Claims 1, 3-6, and 8-10 recite a method for assessing the quality of dataset used in training machine learning model (representing a process);
Claim 11 recite an apparatus (representing a machine); and
Claims 12, 14-17, and 19-20 recite a computer program product (representing an article of manufacture).
Therefore, each set of the claims falls into one of the four statutory categories (i.e., process, machine, article of manufacture, or composition of matter).
Examiner’s note: the specification defines the “storge medium” included in the computer program to exclude signals, see paragraph [0051].
Claims
1, 3-6, 8-12, 14-17 and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more, and hence is not patent-eligible subject matter.
Regarding Amended Claim 1,
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
assessing a quality of the dataset before the dataset is used in building the machine-learning model, (The “assessing” step is an abstract idea of “a mental process”. Examiner’s note: Under the broadest reasonable interpretation (BRI), the step “assessing a quality of the dataset” is a process that can be practically performed in the human mind include observations, evaluations, judgments, and opinions and/or a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation, see MPEP § 2106.04(a)(2)(III). An individual can manually assess the quality of dataset using different metrics and/or characteristics.)
wherein the quality is assessed in view of an effect of the dataset on a performance of the machine-learning model, wherein the assessing comprises scoring the dataset with respect to each of a plurality of attributes of the dataset, wherein the plurality of attributes of the dataset comprises a boundary complexity attribute, a parity attribute, a class overlap attribute, a label purity attribute; (That is part of the abstract idea of the data assessment. The claim merely defines quality metric (attributes) to assess the quality of the dataset. Thus, using different data quality attributes to assess the quality of the dataset is a process that can be practically performed in the human mind with physical aid (e.g., pen and paper). See MPEP § 2106.04(a)(2)(III).)
wherein the boundary complexity attribute indicates a complexity of boundaries that separate classes within the dataset; assessed using an optimization framework that weights different features affecting boundary complexity; (That is part of the abstract idea of mental process. Examiner’s note: assessing the quality of data by measuring the complexity of boundaries that separate classes is also a process that can be performed in the human mind and/or with the aid of a pencil and a paper. Additionally, assessing by weighting different features contribution to the boundary complexity would also fall under the mental process grouping (including an observation, evaluation, judgment, or opinion) (see MPEP § 2106.04(a)(2), subsection III).)
the class overlap attribute is assessed using a label propagation on partial disagreement points approach to identify overlapping class regions; (That is part of the abstract idea of a mental process (i.e., assessing the quality of data with respect to the class overlap attribute). Further, using label to identify overlapping class regions can be practically performed in the human mind with physical aid (e.g., pen and paper). The claim does not define the “label propagation”, thus, it merely represents the method used to identify disagreements where classes overlap.)
and the label purity attribute is assessed by identifying both noisy and confusing data labels within the dataset; (That is part of the abstract idea of a mental process (i.e., assessing the quality of data with respect to the label purity). For example, An individual can manually identify incorrect labels and/or identify images that would be difficult to classify such as blurry images.)
for each of the plurality of attributes having a low-quality score, providing at least one recommendation for increasing the quality of the dataset with respect to the attribute having the low quality-score, wherein the at least one recommendation is based on a specific assessment of the boundary complexity, class overlap, and label purity attributes; (The “providing at least one recommendation” step is an abstract idea of “a mental process”. Examiner’s note: Under the broadest reasonable interpretation (BRI), the “recommending” step is a process that can be practically performed in the human mind including observations, evaluations, judgments, and opinions. See MPEP § 2106.04(a)(2)(III). A data scientist can manually analyze, determine, and provide recommendations based on the data analysis results.)
identify the at least one recommendation and to update the at least one recommendation; (That is part of the abstract idea of a “Mental Process.” The step involves using user input or feedback to “identify and update” recommendation related to the data quality assessment, as recited in the claim, is a process that, under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The process of identifying or changing recommendation is act that would fall under the mental process concepts including an observation, evaluation, judgment, and opinion, see MPEP § 2106.04(a)(2)(III).)
correcting identified noisy and confusing data labels by clustering the identified noisy and confusing data labels based upon the classes defined within the dataset; (This step would fall under the abstract idea of mental process. This involves cleaning dataset by grouping and removing noisy and confusing data label, thus, falls under the mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III). A data scientist can manually identify/remove noisy and confusing labels within a dataset.)
performing a class-wise sample probability and computing a confidence matrix; (This step would fall under the abstract idea of mental process. This involves statistical analysis to compute how confidently each sample belongs to each class, which can be practically performed in the human mind with physical aid (e.g., pen and paper).)
comparing confusing data points identified by the machine learning model and confusing data points identified through clustering; (This step would fall under the abstract idea of mental process. This involves comparing results to assess the quality of the dataset, which is an act of evaluating the results that can be practically performed in the human mind. See MPEP § 2106.04(a)(2)(III).)
aggregating the noisy and confusing data points; (That is part of the abstract idea of a mental process. This step involves grouping noisy and confusing data points, which can be performed in the human mind using physical aid (e.g., pen and paper). See MPEP § 2106.04(a)(2)(III).)
adjusting feature weights to reduce unnecessary boundary complexity; (That is part of the abstract idea of a mental process. This step involves adjusting the scores of different features based on its contribution to the boundary complexity. Thus, adjusting feature weights can be practically performed in the human mind. See MPEP § 2106.04(a)(2)(III).)
applying data transformations to reduce class overlap, (That is part of the abstract idea of a mental process. This step involves applying a mathematical operation to transform data, which can be practically performed in the human mind with physical aid (e.g., pen and paper). See MPEP § 2106.04(a)(2)(III).)
for each of the plurality of attributes having the low-quality score, providing an explanation explaining a cause of the low quality score for the attribute having a low quality score; (The “providing an explanation” step is an abstract idea of “a mental process”. Examiner’s note: Under the broadest reasonable interpretation (BRI), the “explaining” step is a process that can be practically performed in the human mind including observations, evaluations, judgments, and opinions. For example, a data scientist can manually analyze, determine, and explain the quality of dataset i.e., issues in dataset (e.g., outlier, missing values, or class imbalance). See MPEP § 2106.04(a)(2)(III).)
providing a visual representation of one or more data points within the dataset that result in the low quality score; (The “providing a visual representation ...etc.” step is an abstract idea of “a mental process”. Examiner’s note: Under the broadest reasonable interpretation (BRI), the visual representation can be performed in the human mind with the aid of pen and paper. The claim does not specify the technical implementation of the visual representation. Thereby, an individual can manually create a graphical representation to illustrate data points with low quality score. See MPEP § 2106.04(a)(2)(III).)
for each of the plurality of attributes having the low quality score, showing how the one or more data points within the dataset that result in the low quality score can become linearly separable within a new polynomial of degree space. (That is part of the abstract idea of “a mental process.” Examiner’s note: transforming data points into high dimensional to show that data points can be linearly separable is a process that can be performed in the human mind with the aid of pen and paper. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: Under this prong, we evaluate whether the claim recites additional elements that integrate the abstract idea into a practical application by considering the claim as a whole. The judicial exception is not integrated into a practical application.
Additional Elements Analysis:
obtaining a dataset for use in building a machine-learning model; (Amount to no more than adding insignificant extra-solution activity to the judicial exception e.g., mere data gathering in conjunction with the abstract idea - see MPEP 2106.05(g).)
The recitation of “utilizing an optimization framework” is part of the abstract idea of assessing the boundary complexity separating two classes. The claim does not define the optimization framework. Thus, this can be an algorithm and/or computer instructions configured to perform the analysis. Accordingly, this would amount to no more than mere instructions to apply the judicial exception on a computer. See MPEP § 2106.05(f).)
using input, or feedback, from a user to identify the at least one recommendation and to update the at least one recommendation, wherein user input addresses each of the plurality of attributes having the low-quality score; (This merely defines a data receiving step of user input. This amounts to adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g). The generic function of receiving user input to modify or implement data analysis including correcting or modifying the dataset represents mere data gathering in conjunction with the abstract idea (i.e., manually correcting the dataset).)
implementing the at least one user-updated recommendation to provide an enriched dataset; (Amount to the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f). This limitation merely describes the system applies user feedback to improve the dataset, which represents merely invoking a computer as a tool to perform the abstract idea.)
before the dataset is used in building the machine learning model, wherein the quality is assessed in view of an effect of the dataset on a performance of the machine-learning model, …, (Generally linking the use of a judicial exception to a particular technological environment or field of use, see MPEP § 2106.05(h). Examiner notes: the claim limitations merely describe the intended use of the claimed method. For example, the improvement described relates to the abstract idea of the data analysis to clean the dataset prior to using it to train the model.)
wherein implementation of the at least one user-updated recommendation improves the performance of the machine-learning model built using the enriched dataset; …, thereby improving an ability of the machine-learning model to accurately classify the one or more data points. (These statements merely define intended use or field of use. According to MPEP § 2111.02, claim language that merely recites a statement of intended use or result (e.g., “wherein” clauses) does not limit the claim unless it imposes a structural or manipulative requirement. In the present case, the claim limitations do not require any additional acts or steps, rather, they merely describe a desired or expected outcome of using the recited steps of assessing the dataset.)
Step 2B: Under this prong, the claim must include additional elements that amount to significantly more than the judicial exception. These elements must not be well-understood, routine, or conventional in the relevant field. When viewed individually and as an ordered combination, the claim does not include any such additional elements that are sufficient to amount to significantly more (i.e., inventive concept).
Additional Elements Analysis:
As noted above, the implementing step and the use of an optimization framework amount to no more than mere instructions to perform judicial exception on a computer. Mere instruction to apply an exception cannot provide an inventive concept.
The “obtaining a dataset” and “using input, or feedback, from a user” steps amount to mere data gathering in conjunction with the abstract idea. The addition of the data gathering steps cannot provide an inventive concept because they merely represents well‐understood, routine activities. The courts have recognized computer functions such as “storing and retrieving information in memory” and/or “receiving or transmitting data over a network” as well‐understood, routine, and conventional functions. See MPEP § 2106.05(d)(II).
Furthermore, the recitation of “improves the performance of the machine-learning model built using the enriched dataset” and “improving an ability of the machine-learning model to accurately classify the one or more data points” merely describes the intended use of the claimed method, which does not meaningfully limit the claim. The process is primarily directed to assessing the quality of data, and the enriched dataset represents the results of that analysis that is used to build the machine-learning model. Accordingly, the machine learning model merely represents a field of use and the enriched dataset merely define the intended results of the data analysis. See MPEP § 2106.05(h).
Accordingly, when viewed as a whole, the claim is primarily directed to the abstract idea of assessing dataset quality using multiple defined attributes and generating explanation and recommendation based on that analysis. The additional elements, whether considered individually or in combination with the judicial exception, do not integrate the judicial exception into a practical application or amount to significantly more than the abstract idea itself.
Therefore, claim 1 does not recite patent-eligible subject matter.
Regarding Amended Claim 3,
Step 2A Prong 1: Claim 3, which incorporates the rejection of claim 1, recites further limitation such as:
Wherein the assessing a quality of the boundary complexity attribute comprises utilizing an optimization framework that weights different features that affect boundary complexity such that features causing more complex boundaries have lower weights than features causing less complex boundaries. (That is part of the abstract idea. Weighting different features is an abstract idea of “Mental Process”. Weighting different feature based on the boundary complexity is a process of comparing/evaluating information that can be practically performed in the human mind.)
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
The addition of the optimization framework is merely adding a generic computer and/or computer instructions to perform the judicial exception. See MPEP § 2106.05(f).
Step 2B: The additional element does not amount to significantly more than the judicial exception. The same analysis utilized under Step 2A Prong 2 is equally true in Step 2B.
Therefore, claim 3 is ineligible.
Regarding Previously Presented Claim 4,
Step 2A Prong 1: Claim 4, which incorporates the rejection of claim 1, recites further limitation such as:
wherein the class parity attribute indicates an imbalance in classes within the dataset. (That is part of the abstract idea assessing the quality of dataset. Evaluating the distribution of classes or categories in the dataset could be performed by an individual mentally and/or a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 4 is ineligible.
Regarding Original Claim 5,
Step 2A Prong 1: Claim 5, which incorporates the rejection of claim 4, recites further limitation such as:
wherein the assessing a quality of the class parity attribute comprises assessing a plurality of factors contributing to class imbalance and aggregating resulting assessment scores of the plurality of factors to generate the quality score for the class parity attribute. (Abstract idea of a “Mental Process”, see MPEP § 2106.04(a)(2)(III). The steps “assessing” class imbalance, “assigning” scores, and “aggregating” the results, are acts of evaluating/analyzing information/data, and displaying certain results of the collection and analysis. Therefore, these steps are part of the data assessment which can be practically performed in the human mind and/or a human using physical aid (e.g., pen and paper or a slide rule) to perform the claim limitations.)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 5 is ineligible.
Regarding Previously Presented Claim 6,
Step 2A Prong 1: Claim 6, which incorporates the rejection of claim 1, recites further limitation such as:
wherein the class overlap attribute indicates an amount of overlap between classes within the dataset. (That is part of the abstract idea assessing the quality of dataset. Evaluating the overlapping distribution of two classes within the dataset can be performed by an individual mentally or with physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 6 is ineligible.
Regarding Previously Presented Claim 8,
Step 2A Prong 1: Claim 8, which incorporates the rejection of claim 1, recites further limitation such as:
wherein the label purity attribute indicates an accuracy of labels within the dataset. (That is part of the abstract idea. An individual (e.g., a data scientist) can assess the accuracy of different class labels within a dataset. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 8 is ineligible.
Regarding Original Claim 9,
Step 2A Prong 1: Claim 9, which incorporates the rejection of claim 8, recites further limitation such as:
wherein the assessing a quality of the label purity attribute comprises identifying both noisy and confusing data labels within the dataset. (That is part of the abstract idea of claim 8. The claim merely defines the assessment by identifying noisy and confusing data labels. This step can be practically performed in the human mind. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 9 is ineligible.
Regarding Original Claim 10,
Step 2A Prong 1: Claim 10, which incorporates the rejection of claim 1, recites further limitation such as:
wherein the providing an explanation comprises identifying at least one data point within the dataset causing the low quality score. (That is part of the abstract idea of claim 1. The claim merely defines the explanation as identifying the data point (i.e., mislabeled data) which is a process that can be practically performed by a human, see MPEP § 2106.04(a)(2)(III). An individual (e.g., data scientist) can manually identify missing data points and/or mislabeled data.)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 10 is ineligible.
Regarding Amended Claim 11,
The claim recites similar limitations as corresponding claim 1. Therefore, the same analysis (subject matter eligibility analysis) that was utilized for claim 1, as described above, is equally applicable to claim 11. The only difference is that claim 1 is drawn to a method, and claim 11 is drawn to an apparatus. The recitation of “an apparatus, comprising: at least one processor; and a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor …,” which is directed to the applying of mere instructions (on a generic computer) to implement a judicial exception, and hence the claimed additional elements listed above are merely generic elements and the implementation of the elements merely amount to no more than instruction to apply the abstract idea using a generic computer component. Therefore, the additional elements do not integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Therefore, claim 11 is ineligible.
Regarding Amended Claim 12,
The claim recites similar limitations as corresponding claim 1. Therefore, the same analysis (subject matter eligibility analysis) that was utilized for claim 1, as described above, is equally applicable to claim 12. The only difference is that claim 1 is drawn to a method, and claim 12 is drawn to a computer program product. The recitation of “computer program product, comprising: a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor…,” merely defines computer component and instructions to implement a judicial exception, and hence the claimed additional elements listed above are merely generic elements and the implementation of the elements merely amount to no more than instruction to apply the abstract idea using a generic computer component. Therefore, the additional elements do not integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Therefore, claim 12 is ineligible.
Regarding Amended Claim 14,
The claim recites similar limitations as corresponding claim 3. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 3, as described above, is equally applicable to claim 14.
Therefore, claim 14 is ineligible.
Regarding Previously Presented Claim 15,
The claim recites similar limitations as corresponding claim 4. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 4, as described above, is equally applicable to claim 15.
Therefore, claim 15 is ineligible.
Regarding Original Claim 16,
The claim recites similar limitations as corresponding claim 5. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 5, as described above, is equally applicable to claim 16.
Therefore, claim 16 is ineligible.
Regarding Previously Presented Claim 17,
The claim recites similar limitations as corresponding claim 6. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 6, as described above, is equally applicable to claim 17.
Therefore, claim 17 is ineligible.
Regarding Previously Presented Claim 19,
The claim recites similar limitations as corresponding claim 8. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 8, as described above, is equally applicable to claim 19.
Therefore, claim 19 is ineligible.
Regarding Original Claim 20,
The claim recites similar limitations as corresponding claim 9. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 9, as described above, is equally applicable to claim 20.
Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 3-6, 8-12, 14-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Maughan et al., (Pub. No.: US 20170372232 A1) in view of Lorena et al., (NPL.: “How Complex Is Your Classification Problem? : A Survey on Measuring Classification Complexity.” (2019)), further in view of Devin et al., (Pub. No.: US 20140247978 A1), further in view of Nicholson et al., (NPL.: “Label noise correction methods.” (2015)), further in view of Hall et al., (Pub. No.: US 20230162049 A1), further in view of Weston et al., (Pub. No.: US 20050216426 A1), further in view of Tung et al., (NPL.: “Reducing class overlapping in supervised dimension reduction.” (2018)), and further in view of Melvin et al., (Pub. No.: US 20180150635 A1).
Regarding Amended Claim 1,
Maughan discloses the following:
A method, comprising: obtaining a dataset for use in building a machine-learning model; (Maughan, [0002] “illustrates a method/system relates to data quality and more particularly relates to detection of and compensation for data quality as an input for machine learning. Maughan [0031] “In general, the predictive analytics apparatus 102 generates and/or executes machine learning models for one or more clients using data from one or more data sources 104.”)
assessing a quality of the dataset before the dataset is used in building the machine learning model, (Maughan, Fig. 1, elements 102 (Predictive Analytics Apparatus) and 104 (dataset), [0004] “Apparatuses are presented for data quality detection and compensation for machine learning. In one embodiment, a quality analysis module electronically identifies one or more data quality issues in machine learning training data.” [0051] “For example, in certain embodiments, the quality analysis module 202 and the corrective action module 204 may automatically pre-process training and/or workload data from data sources 104 prior to the predictive analytics module 206 creating and/or applying a machine learning model.” [Note: involve pre-processing data prior to creating or applying to machine learning model.]) wherein the quality is assessed in view of an effect of the dataset on a performance of the machine-learning model, (Maughan, [0049] “A data quality issue may refer to any problem, or potential problem, that could affect the performance of the predictive analytics module 206 in creating or executing a machine learning model. Certain types of data quality issues may cause the predictive analytics module 206 to fail to build a machine learning model, to build an inaccurate machine learning model (e.g., due to overfitting, or the like).”)
wherein the assessing comprises scoring the dataset with respect to each of a plurality of attributes of the dataset, (Maughan, [0051] “The quality analysis module 202 may determine or calculate attributes of various features, such as a number or percentage of unique values for a feature, a number or percentage of missing values or outliers for a feature, a mean, variance, or standard deviation of numerical values for a feature, or the like. In certain embodiments, the quality analysis module 202 may compare determined or calculated attributes to one or more predetermined thresholds to determine whether (or to what extent) a data quality issue exists. In one embodiment, thresholds for determining whether (or to what extent) a data quality issue exists may be set by a manufacturer of the predictive analytics apparatus 102. In another embodiment, thresholds for determining whether (or to what extent) a data quality issue exists may be set or updated by a user, such as a data scientist.” Further described in [0110] – [0112]. The model-readiness module 302, in one embodiment, is configured to provide one or more model-readiness scores to a user based on the one or more data quality issues identified by the quality analysis module 202.” [Note: the percentage that is determined based on the calculated attributes of different dataset would represent the scoring aspect. Further, Maughan teaches the quality analysis including scoring features to address quality issues.]), wherein the plurality of attributes of the dataset comprises …, a class parity attribute, …, and a label purity attribute; (Maughan, [0080] “The quality analysis module 202 processes data to detect a balance/distribution of label/target/dependent variables, missing values in label/target/dependent variables, or the like. The quality analysis module 202, in certain embodiments, may detect an unbalanced label, …etc.” [0081] “The quality analysis module 202 may score numeric features based on how “normal” their distribution is, how well the feature fits a common distribution, or the like.” [0094] “For example, the quality analysis module 202 may determine that a size and/or ratio of one or more classes or sets has changed and/or drifted over time, or the like. In one embodiment, the quality analysis module 202 may monitor and/or analyze confidence metrics from the machine learning model.” [Note: Maughan includes multiple attributes used to measure/analyze the quality level of dataset for example class imbalance and label purity (label consistency and drifts in the ratio of the classes).]) and wherein: …, the label purity attribute is assessed by identifying both noisy and confusing data labels within the dataset; (Maughan, [0080] “the quality analysis module 202 processes data to detect a balance/distribution of label/target/dependent variables, missing values in label/target/dependent variables, or the like. The quality analysis module 202, in certain embodiments, may detect an unbalanced label, …” [0085] “The quality analysis module 202 may identify and score features for unexpected value types, …” Furter see [0089]. [Note: imbalance label, labels/outputs that drift, distribution shifts, anomalous label values or missing values in labels.])
for each of the plurality of attributes having a low quality score, providing at least one recommendation for increasing the quality of the dataset with respect to the attribute having a low quality score; (Maughan, [0072] “… The quality analysis module 202, in one embodiment, may detect one or more unique identifiers in data, and the corrective action module 204 may recommend and/or take one or more associated corrective actions.” [0077] “The quality analysis module 202, in one embodiment, may process data to detect one or more of positive and/or negative infinity. Positive and negative infinite values, in certain embodiments, may cause problems in modeling. The quality analysis module 202 may provide a score by feature and/or by dataset comprising or based on a percentage and/or number of infinite values. The corrective action module 204, in various embodiments, may perform and/or recommend performance of replacing infinite values with high or low non-infinite values, treating an infinite value as missing, excluding a feature vector with one or more infinite values.” [0053] “A corrective action may be based on, or in response to, a data quality issue if the corrective action corrects, improves, or otherwise affects the data quality issue.” Further see [0075]-[0076]. [Note: Maughan provides recommendations to address data quality issues such as unbalance label and missing values, label distribution, drift in the label/target or the like.]) wherein the at least one recommendation is based on a specific assessment of the boundary complexity, the class overlap, and the label purity attributes; (Maughan, [0085] “The quality analysis module 202, in certain embodiments, may process data to detect one or more unusual and/or unexpected values (e.g., data types) within a feature vector, and the corrective action module 204 may perform and/or recommend an associated corrective action.”[0089] “The quality analysis module 202, in certain embodiments, may process data to detect drift. …, Drift may occur in the design matrix or in the label/target. In response to detecting drift, the corrective action module 204 may perform and/or recommend a corrective action such as excluding the features with drift above a threshold, repairing the features with drift by imputing and/or transforming values, or the like.” [Note: the recommendations generated by the corrective action module are based on the analysis of different data quality issue attributes performed by the quality analysis module.])
using input, or feedback, from a user to identify the at least one recommendation and to update the at least one recommendation, wherein user input addresses each of the plurality of attributes having the low quality-score; implementing the at least one user-updated recommendation to provide an enriched dataset, wherein implementing comprises: (Maughan, [0056] “In one embodiment, the corrective action module 204 may apply one or more different corrective actions (e.g., different from the corrective actions applied to training data) to workload data, in response to user input. The corrective action module 204 may provide an interface allowing a user to change or update which corrective actions are taken over time, and may cooperate with the predictive analytics module 206 102 to update a model for the data, in certain embodiments, in response to a user changing or updating which corrective actions are taken of the data (e.g., in response to determining that a change or update requires a change in an associated model, or the like).” [0058] “The corrective action module 204 may be configured to perform certain corrective actions automatically and others in response to user input, to perform all corrective actions automatically, or the like. ….etc.” [0059] “the corrective action module 204 determines the one or more corrective actions based on a quality level selected by a user. .... a quality level may indicate a user preference without specifying particular corrective actions, and the corrective action module 204 may select the corrective actions to perform based on the quality level.” [0060] “the corrective action module 204 may apply different corrective actions for different model or algorithm types used by the predictive analytics module 206, to create multiple versions of modified training data for the different algorithm types.” Further see [0112]-[0116].) [Examiner’s Note: the system identifies data quality issues and generates recommended corrective actions. These recommendations are presented to the user via an interface (e.g., GUI), allowing the user to review, select, and/or update corrective actions. The system then applies the selected or user-updated corrective actions to the dataset, providing modified dataset for subsequent machine learning or predictive analytics.]
for each of the plurality of attributes having a low quality score, providing an explanation explaining a cause of the low quality score for the attribute having a low quality score, (Maughan, [0058] “The corrective action module 204 may provide a user an automated corrective action after it is taken (e.g., providing with the notification and explanation of the automated corrective action, or the like). The corrective action may recommend and/or take one or more associated corrective actions. For example, a unique id feature comprising only unique identifiers may be apart of a dataset, but often should not be included in modeling.” [0072] “The corrective action module 204 may present likely IDs together with one or more reasons why they might be IDs, and may ask a user to flag them as IDs to confirm the quality analysis module 202's determination, or the like.” [0115] “The GUI module 304 may allow a user to authorize or deny performance of one or more recommended corrective actions (e.g., fixes or the like). In certain embodiments, the GUI module 304 may provide automatic and/or single-click recommended fixes to the data along with explanations of the errors and the fixes. For example, the GUI module 304 may provide a user with a list of one or more recommended corrective actions, and may prompt the user to accept all recommended corrective actions, to cancel or deny all recommended corrective actions, or the like.”) and providing a visual representation of one or more data points within the dataset that result in the low quality score; (Maughan, [0058] “a score for a feature of the training data, a score for a dependent variable, a score for a potential data quality issue, and/or the like. Various scores discussed above for different data quality issues may be presented to a user by the model-readiness module.” [0112]-[0116] “The GUI module 304, in one embodiment, is configured to interactively presents the one or more data quality issues identified by the quality analysis module 202, and one or more potential corrective actions to a user. A “potential” corrective action may be a possible or recommended corrective action, and the one or more corrective actions actually performed by the corrective action module 204 may be selected by the user from the one or more potential corrective actions.” Further see [0107] “display to the user …etc.”.)
.... thereby improving an ability of the machine-learning model to accurately classify the one or more data points. (Maughan, [0060] “…, the corrective action module 204 may apply different corrective actions for different model or algorithm types used by the predictive analytics module 206, to create multiple versions of modified training data for the different algorithm types. In various embodiments, tailoring corrective actions 204 to particular machine learning algorithms may increase the accuracy or efficiency of machine learning models generated by the predictive analytics module 206.” Further described in [0102].)
As noted above, Maughan teaches a quality analysis module that is used to identify and evaluate quality issues within a dataset used to train a machine learning model, using different attributes or measures.
Maughan does not appear to explicitly teach the following:
assessing a boundary complexity attribute, wherein the boundary complexity attribute indicates a complexity of boundaries that separate classes within the dataset, assessed using an optimization framework that weights different features affecting boundary complexity;
assessing a class overlap attribute, wherein the class overlap attribute is assessed using a label propagation on partial disagreement points approach to identify overlapping class regions;
wherein implementing comprises: correcting identified noisy and confusing data labels by clustering the identified noisy and confusing data labels based upon the classes defined within the dataset; performing a class-wise sample probability and computing a confidence matrix; comparing confusing data points identified by the machine learning model and confusing data points identified through clustering; aggregating the noisy and confusing data points; adjusting feature weights to reduce unnecessary boundary complexity; and applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Lorena, in combination with Maughan, teaches:
assessing a boundary complexity attribute, wherein the boundary complexity attribute indicates a complexity of boundaries that separate classes within the dataset, (Lorena, [Pp. 3 & 11; section 2 & 2.3.1] discloses, “The complexity of a classification problem can be attributed to a combination of three main factors: (i) the ambiguity of the classes; (ii) the sparsity and dimensionality of the data; and (iii) the complexity of the boundary separating the classes.” [107: 3] In this article, we group the complexity measures into more categories, as follows:” (six categories, see Lorena page 3).) assessed using an optimization framework that weights different features affecting boundary complexity; (Lorena, [107: 3, Section: 2.1] teaches: “Feature-based measures, which characterize how informative the available features are to separate the classes; ….. 2.1 Feature-based Measures: These measures evaluate the discriminative power of the features. In many of them, each feature is evaluated individually. If there is at least one very discriminative feature in the dataset, the problem can be considered simpler than if there is no such attribute. All measures from this category require the features to have numerical values. Most of the measures are also defined for binary classification problems only.” [107: 4] “The first measure presented in this category is the maximum Fisher’s discriminant ratio, denoted by F1. It measures the overlap between the values of the features in different classes … where
r
f
i
is a discriminant ratio for each feature
f
i
. Originally, F1 takes the value of the largest discriminant ratio among all the available features. …, Herewith, the F1 values become bounded in the (0, 1] interval and higher values indicate more complex problems, where no individual feature is able to discriminate the classes.” See (Eq. 2 and Eq. 3). Moreover, Lorena [107: 9, Section: 2.2] teaches: Measures of Linearity “The hyperplane sought in the SVM formulation is the one that separates the examples from different classes with a maximum margin while minimizing training errors. …., Low values for L1 (bounded in [0, 1)) indicate that the problem is close to being linearly separable—that is, simpler.” Further, Lorena [107: 11, Section: 2.3] teaches: Neighborhood Measures “N1 estimates the size and complexity of the required decision boundary through the identification of the critical points in the dataset: those very close to each other but belonging to different classes. Higher N1 values indicate the need for more complex boundaries to separate the classes and/or that there is a large amount of overlapping between the classes. N1 can be expressed as: [see equation (22)].”) [Examiner note: Lorena teaches feature-based measure such as using ratio that quantities individual features contributing to the complexity in the classification problem separating classes. This would read on the weights different features affecting boundary complexity. The optimization framework is broadly interpreted as the feature-based measure technique including using SVM classifier.]
wherein assessing a class overlap attribute; (Lorena, [107: 11] “2.3 Neighborhood Measures: These measures try to capture the shape of the decision boundary and characterize the class overlap by analyzing local neighborhoods of the data points.” [107: 19] “Finally, the class density in overlap region (D3) determines the density of each class in the overlap regions. It counts, for each class, the number of points lying in the same region of a different class.”) wherein the class overlap attribute is assessed using a label propagation on partial disagreement points approach to identify overlapping class regions; (Lorena, [107:7] “For each feature, it checks whether there is overlap of values between examples of different classes. If there is overlap, the classes are considered to be ambiguous in this region. The problem can be considered simpler if there is at least one feature that shows low ambiguity between the classes, …., where no (fi ) gives the number of examples that are in the overlapping region for feature fi and can be expressed by Equation (11). Low values of F3, computed by Equation (10), indicate simpler problems, where few examples overlap in at least one dimension.” [107: 17] “In complex datasets, in which a high overlapping of the classes is observed, strong vertexes will tend to be less connected to strong neighbors. However, for simple datasets there will be dense regions within the classes and higher hub scores.”) [Note: Lorena teaches multiple class overlap measures to identify overlapping regions. This involves using ϵ-NN, unsupervised and supervised learning. See Fig. 12.]
wherein the at least one recommendation is based on a specific assessment of the boundary complexity, the class overlap, and the label purity attributes; (Maughan, [0085] “The quality analysis module 202, in certain embodiments, may process data to detect one or more unusual and/or unexpected values (e.g., data types) within a feature vector, and the corrective action module 204 may perform and/or recommend an associated corrective action.”[0089] “The quality analysis module 202, in certain embodiments, may process data to detect drift. …, Drift may occur in the design matrix or in the label/target. In response to detecting drift, the corrective action module 204 may perform and/or recommend a corrective action such as excluding the features with drift above a threshold, repairing the features with drift by imputing and/or transforming values, or the like.”) [Note: the recommendations generated by the corrective action module are based on the analysis of different data quality issue attributes performed by the quality analysis module.]
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, to modify the method/system of Maughan to incorporate the categories of complexity measures (used to solve classification problems within training dataset in the machine learning) as taught by Lorena. One would have been motivated to make such combination in order to effectively estimate the difficulty in separating the data points into their expected classes. This information can support the formulation of new data-driven pre-processing and pattern recognition techniques, which can in turn be focused on challenges highlighted by such characteristics of the problems, and analyzes measures which can be extracted from the training datasets in order to characterize the complexity of the respective classification problems (Lorena, [Abstract & Conclusion]).
As outlined above, Lorena discloses a method for assessing the complexity of a dataset by evaluating the extent to which one or more data points are linearly separable. Lorena also describe the measure that calculates overlapping regions. Lorena discusses and motivates the use of SVM techniques for feature transformation to map data into a high-dimensional space where linear separability is more likely (See Pp. 9 and 29). Additionally, while Maughan teaches providing recommendation to the user for data quality issues, receiving user feedback to select or update corrective actions, and applying (i.e., implementing) corrective actions to generate a modified dataset, Maughan in view of Lorena does not appear to explicitly suggest:
wherein implementing comprises: correcting identified noisy and confusing data labels by clustering the identified noisy and confusing data labels based upon the classes defined within the dataset; performing a class-wise sample probability and computing a confidence matrix; comparing confusing data points identified by the machine learning model and confusing data points identified through clustering; aggregating the noisy and confusing data points; adjusting feature weights to reduce unnecessary boundary complexity; and applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Devin, in combination with Maughan in view of Lorena, teaches the following:
correcting identified noisy and confusing data labels by clustering the identified noisy and confusing data labels based upon the classes defined within the dataset; [Examiner’s note: As established earlier, this limitation exhibits a 112(a) written description issue, and hence for the purposes of examination, this limitation is interpreted in light of the Spec broadly as clustering data points based upon the classes defined within the dataset, identifying noisy/confusing data points, and correcting the data label.] (Devin, [0032] “... The present system applies one or more heuristics 24 to evaluate the labeled training set 10. In particular, clustering techniques, based on the score vectors provided by the broad categorizer, may be used to analyze the training data set 10 and identify deficiencies. These may be used to provide automatic enhancements to the labeled training set 10, e.g., by modifying some of the labels and/or to propose modifications.” [0036]-[0038] “The exemplary instructions may include the broad categorizer 14, a representation generator 130, a clustering component 132, a metrics component 134, an evaluation component 136, a recommendation component 138, and a custom classifier training component 140. ... The clustering component 132 identifies clusters of training images based on their score vectors 22 (e.g., at least two, each cluster including a set of score vectors assigned to the cluster).” [0049]-[0052] “At S106, the training data is screened with the broad categorizer 14 to generate a score vector 22 for each digital object in the labeled training set 10. If a Universe Representative Set (URS) 12 is provided, score vectors 22 for the digital objects in the URS may also be generated. At S108, objects may be clustered based on their scores, by the clustering component 132. FIG. 4 illustrates an example of a cluster, which is described in greater detail below. Metrics (statistics) are computed based on the clusters. At S110 heuristics are applied to the metrics computed at S108 to evaluate the training set and to determine deficiencies in the labeled and unlabeled training data 10, 12. At S112, modifications to the training data 10, 12 are proposed and/or implemented automatically. In the exemplary embodiment, the recommendation component 138 may output a recommendation, based on the results from S110. The recommendation may include an alert, a warning, and/or a suggestion, which may alert the submitter to possible inconsistencies in the submitted data and/or make suggestions for its modification. The system may automatically label (or remove labels from) the training data in order to optimize the quality of the resulting classifier 16. Provision may be made for the user to modify the training data.” [0077] The noise threshold may function as a preliminary warning. If less than a predetermined proportion (e.g., 50%) of the score vectors 22 are above the noise threshold, this implies that the classes being trained may be outside the scope of the existing categories for the existing broad categorizer 14. Few scores above a noise threshold may also be a sign of low visual consistency among the digital objects (e.g., images).” Further described in [0087].) [Note: the low consistency and/or overlapping data points interpreted as the confusing/noisy points.]
Accordingly, at the effective filing date of the claimed invention, it would have been prima facie obvious to one of ordinary skill in the art to modify the combination of Maughan and Lorena to incorporate the method for evaluating training data as taught by Devin. One would have been motivated to make such a combination in order to provide automatic enhancements to the training set, e.g., by modifying some of the labels and/or to propose modifications. Doing so would improve the training data to address the identified deficiencies, which are output from the system (Devin [0032]-[0039]).
The combination of Maughan, Lorena, and Devin does not appear to explicitly teach:
performing a class-wise sample probability and computing a confidence matrix; comparing confusing data points identified by the machine learning model and confusing data points identified through clustering; aggregating the noisy and confusing data points; adjusting feature weights to reduce unnecessary boundary complexity; and applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Nicholson, in combination with Maughan, Lorena, and Devin, teaches the following:
identifying noisy/confusing data points, and correcting the noise label, comparing confusing data points identified by the machine learning model and confusing data points identified through clustering; [Examiner’s note: As established earlier, this limitation exhibits a 112(a) written description issue, and hence for the purposes of examination, this limitation is interpreted in light of the Spec broadly as comparing confusing/noisy data points identified by an algorithm and the confusing/noisy point identified through the clustering.] (Nicholson, [P.1, Section: I, Col. 2] “... there is only one method to correct noisy labels (belonging to a discrete set)— that is, to identify each mislabeled instance and to correct it according to what a correction method deems its most likely ground-truth label (i.e., its true class) to be.” [P. 2, Section II-B and II-C] “In specific, the Self-Training Correction algorithm first uses a noise-filtering algorithm on a data set to generate a noisy set and a clean set. Then, the algorithm builds a model from the clean set and uses that to calculate the confidence that each of the instances from the noisy set is mislabeled. The noisy instance with the highest calculated likelihood of belonging to some class that is not equal to its current class is relabeled to the class that the classifier determined is the instance’s most likely true class. The relabeled instance is then added to the clean set, and the process is repeated until some proportion of noisy instances is relabeled and added to the clean set. SelfTraining Correction is flexible in that the user can specify what proportion of noisy instances he wants to correct. .... Our cluster-based correction method (denoted as CC) executes one or more clustering algorithms on a training set several times, giving the same set of weights to all instances in each cluster based on the distribution of ascribed labels in that cluster and the cluster’s size. The weights favor the class where the majority (or plurality) of instances belong to that class. After the data set has been clustered several times, weights obtained from each cluster are summed for each instance, and each instance is given the label corresponding to the maximum weight.” [P. 2, Section: I] “In Section 2, we begin by discussing two correction methods that we developed(Self-Training Correction and Cluster-based Correction), and another method we adapted to compare with our methods (Polishing Labels). In Section 3, we test these three methods in their ability to improve label quality, model quality, and AUC on binary class data sets and multi-class data sets. In Section 4, we conclude that Clusterbased Correction achieves the best, most consistently positive results.” [P. 3, Section: III] “we describe our experimental setup and provide our experimental results for comparing the three methods in terms of label quality, model quality, and AUC. In multi-class scenarios, AUC was calculated using a pairwise average AUC metric over each pair of classes.” ) [Note: Nicholson describes the experiments involving comparing correction methods in terms of label quality, model quality, and AUC. These methods are used to identify and correct label noise.]
Therefore, it would have been prima facie obvious to one of ordinary skill in the art, before the effective date of the claimed invention, having the combination of Maughan, Lorena, and Devin, to incorporate the Label Noise Correction Methods as taught by Nicholson. One would have been motivated to make such a combination in order to improve label quality, model quality, and AUC on binary class data sets and multi-class data sets (Nicholson [Section: IV]).
While, the combination of Maughan, Lorena, Devin, and Nicholson teaches the concept of implementing user-updated recommendation such as identifying overlapping between two label groups and noisy instance, correcting/modifying deficiencies in the training data, and comparing the result of different algorithms, the combination of Maughan, Lorena, Devin, and Nicholson does not appear to explicitly teach:
performing a class-wise sample probability and computing a confidence matrix; ... aggregating the noisy and confusing data points; adjusting feature weights to reduce unnecessary boundary complexity; and applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Hall, in combination with Maughan, Lorena, Devin, and Nicholson, teaches the following:
performing a class-wise sample probability and computing a confidence matrix; (Hall, [0177] “ In classification problems, the calculation of the cross entropy loss, which compares a one-hot encoded (true) probability pj (c) distribution over classes c∈{1 . . . C} to the estimated probability distribution qj (c) for each element j∈{1 . . . N}. ... Class-based accuracy A(c): is valuable when one would like to see the correct prediction rate per class (NT (c)). The calculation is like the accuracy, but we only consider images of one class (c) at a time: A (c) =N T (c) /N (c) (3) ...” [0182] “Confidence based metrics include Log loss, combined class Log loss, combined data-source Log loss, combined class and data-source Log loss.” [0193] “The metrics may also be a confidence based metric such as Log Loss (which may be used as a primary metric).” [0194] “The AI model's classification (or prediction) for each sample (zj s) in the training dataset can be compared with its assigned label/class to determine if the sample was correctly or incorrectly classified.”)
aggregating the noisy and confusing data points; (Hall, [0198] “In the case where there are multiple datasets from multiple data-sources (or data owners), perform the data cleansing in (a) above to each sub-dataset, enabling the removal or re-labeling of mis-labeled samples from each sub-dataset. Aggregate the multiple sub-datasets for machine learning training. An optional step is to perform the data cleansing in (a) above again on the aggregated dataset to remove any remaining mis-labeled samples. Finally train machine learning models on the aggregated and cleansed dataset.” [0317] “Accumulate the 5 output files, and only include images from the Noisy Class (as these are the only images that are assumed to be potentially mis-labeled), to produce a single output file which contains for each (non-viable) image in the dataset: (1) the number of models that produce incorrect results (maximum of 5); and (2) the mean incorrect prediction score of these models. The mean prediction score indicates how far these models are getting wrong predictions. A short list of images was created that included (non-viable) images that were misclassified by multiple models, say 4 or 5 models, with high incorrect prediction scores. These images are considered as the mis-labeled data and are candidates for removal or re-labeling. The “mis-labeled” images in the list were removed from the aggregated training set in order to cleanse the dataset. [0323] The following experiment was used to compare the validation and test results of models which were trained on the original training set and on the cleaned training set (removed list of mis-labeled images as described above). The metric used to assess the results was balanced accuracy, but other metrics, such as confidence based metrics (e.g. Log Loss) could have been used.” [0327] “Create a short list of images (only in the Noisy Class, i.e. the non-viable class) that were misclassified in each file. The predicted scores can be used for a thresholding filter purpose. Remove these images from the training dataset and then aggregate all the datasets and re-train the best models on the new cleaned aggregated dataset.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, having the combination of Maughan, Lorena, Devin, and Nicholson to incorporate the method for cleaning a dataset for generating an Artificial Intelligence (AI) model as taught by Hall. One would have been motivated to make such a combination in order to address/minimize the label noise by cleansing the dataset and removing any mis-labeled or noisy data. Doing so would ultimately improve the data quality and thus the AI model performance (Hall [0360]).
The combination of Maughan, Lorena, Devin, Nicholson, and Hall does not appear to explicitly teach:
adjusting feature weights to reduce unnecessary boundary complexity; and applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Weston, in combination with Maughan, Lorena, Devin, Nicholson, and Hall, teaches the following:
adjusting feature weights to reduce unnecessary boundary complexity; (Weston, [0078] “A simple feature ranking can be produced by evaluating how well an individual feature contributes to the separation (e.g. cancer vs. normal). Various correlation coefficients have been proposed as ranking criteria.” [0082]-[0087] “One aspect of the present invention comprises using the feature ranking coefficients as classifier weights. ... RFE methods comprise iteratively 1) training the classifier , 2) computing the ranking criterion for all features, and 3) removing the feature having the smallest ranking criterion. ... the weights of a classifier are used to produce a feature ranking with a SVM (Support Vector Machine). The present invention contemplates methods of SVMs used for both linear and non-linear decision boundaries of arbitrary complexity, however, the example provided herein is directed to linear SVMs because of the nature of the data set under investigation. Linear SVMs are particular linear discriminant classifiers. (See Equation 1). If the training set is linearly separable, a linear SVM is a maximum margin classifier. The decision boundary (a straight line in the case of a two-dimension separation) is positioned to leave the largest possible margin on either side.” [0203] “These features can then be used for another classifier such as a SVM. If there are still too many features, they can be ranked according to the absolute value of the weight vector of coefficient assigned to them by the separating hyperplane.” Further described in [0261].) [Examiner’s Note: the limitation of “adjusting feature weights to reduce unnecessary boundary complexity” broadly interpreted as ranking features based on their contribution to the decision boundary (hyperplanes) that separate classes.]
While, Weston, in combination with Maughan, Lorena, Devin, Nicholson, and Hall, teaches applying pre-processing transformation step (linear and non-linear), the combination of Maughan, Lorena, Devin, Nicholson, Hall, and Weston does not explicitly teach:
applying data transformations to reduce class overlap,
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Tung, in combination with Maughan, Lorena, Devin, Nicholson, Hall, and Weston, teaches:
applying data transformations to reduce class overlap, (Tung, [P. 9, Section: 1, Col. 1] “In this paper, we propose a new method that extends the SDR framework by keeping the neighborhood graphs and the class overlapping separated from each other. The novelty of our method is that our extension of the SDR framework reduces the size of the class overlapping, while still inheriting the advantages of the original framework. As shown by our numerical experiments, we reduce the class overlapping ’s cardinality by an order of magnitude. As a result, we improve the accuracy of the framework on the classification task significantly. Moreover, the new reduced-dimensional representation of the dataset is not only more separated for different classes but also more scattered for the same class, as compared to the original representation.” [Pp. 10-11, Section: 3] “In this section, we propose a new method that reduces the overlap in most scenarios. The new method is specified in Algorithm 2. The iterative algorithm contains 4 main steps. First, we find the nearest neighbors {d ′ } which lie outside the overlap areas O for each data point d, in order to dilute/reduce the overlap zones O. Second, we infer θ ∗ d for each data point d by optimizing the objective function J(d; θ, β). Third, we learn a lower-dimensional space B∗ from all θ ∗ d and D ′ to project the original dataset D into using step 4 of the SDR framework. Fourth, we use SVM classification with 5-fold cross-validation to approximate the overlap regions O. The main steps of the algorithm and its comparison to Algorithm 1 is specified in more details in the following subsections.”) [Examiner’s Note: applying dimensionally reduction (data transformation) to reduce class overlap.]
The combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, and Tang does not appear to explicitly teach:
showing how the one or more data points within the dataset that result in the low-quality score can become linearly separable within a new polynomial of degree space,
However, Melvin, in combination with Maughan, Lorena, Devin, Nicholson, Hall, Weston, and Tang, teaches:
showing how the one or more data points within the dataset that result in the low quality score can become linearly separable within a new polynomial of degree space. (Melvin, [0008] “FIGS. 5a and 5b demonstrate how a linear boundary can be created with complex data by projecting it to a higher dimensional space.” [0021] “Occasionally, real world data is not always linearly separable by a classifier or hyperplane. This presents a challenge to linear classifiers such as the Support Vector Machines to separate data reliably. However, as mentioned earlier, by mapping the low dimensional data onto a space of sufficiently higher dimension, a linear separation between the competing classes can be found and therefore can be separated using a hyperplane. FIG. 5a shows complex data in low dimensions, and FIG. 5b shows that complex data being turned into separable data in a higher dimension, or an infinite dimensional space produced by the RBF kernels, where it can be separated and used in a hyperplane.” Further see [0018].) [Note: projection to higher dimensional space (polynomial feature space) i.e., using Kernel function to transform data to dimensional space where a linear hyperplane can separate classes.]
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, and Tang, before them, to incorporate the SVMs using the kernel methods to transform complex data into a higher dimensional space as taught by Melvin. One would have been motivated to make such a combination in order to enable linear separation of complex data where data may not be linearly separable in its original lower-dimensional form (Melvin, [0021]).
Regarding Amended Claim 3, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
Weston further teaches: wherein the optimization framework weights different features that affect boundary complexity such that features that cause unnecessarily more complex boundaries have lower weights than features causing less complex boundaries. (Weston, [0082] “One aspect of the present invention comprises using the feature ranking coefficients as classifier weights. Reciprocally, the weights multiplying the inputs of a given classifier can be used as feature ranking coefficients. The inputs that are weighted by the largest values have the most influence in the classification decision. Therefore, if the classifier performs well, those inputs with largest weights correspond to the most informative features, or in this instance, genes.” [0261] “The goal is to define a linear model that minimizes the Ranking Loss while having a low complexity. The notion of complexity is the margin. For systems that rank the values of <wk, x>+bk, the decision boundaries for x are defined by the hyperplanes whose equations are <wk−wl, x>+bk−bl=0, where k belongs to the label sets of x while l does not.”) [Examiner’s Note: The SVM using RFE (optimization framework) that ranks features based on the influence on decision boundaries. Features that are less informative (i.e., unnecessarily) receive less weights or eliminated, while more informative features ranked higher weights.]
Regarding Previously Presented Claim 4, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
wherein the class parity attribute indicates an imbalance in classes within the dataset. (Maughan, [0080] “The quality analysis module 202 processes data to detect a balance/distribution of label/target/dependent variables, missing values in label/target/dependent variables, or the like. The quality analysis module 202, in certain embodiments, may detect an unbalanced label, ..., The corrective action module 204, in various embodiments, may perform a corrective action comprising weighting the classes, sampling to increase the balance, transforming a numeric distribution or recommend excluding unusual values, or the like.” [0094] “For example, the quality analysis module 202 may determine that a size and/or ratio of one or more classes or sets has changed and/or drifted over time, or the like.”)
Regarding Original Claim 5, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 4 as outlined above, and further teaches:
Maughan further teaches: wherein the assessing the quality of the class parity attribute comprises assessing a plurality of factors contributing to class imbalance and aggregating resulting assessment scores of the plurality of factors to generate the quality score for the class parity attribute. (Maughan, [0081] “The quality analysis module 202 may process data to detect an unusual distribution and may score numeric features based on how “normal” their distribution is, how well the feature fits a common distribution, or the like. The corrective action module 204, in various embodiments, may perform and/or recommend one or more transformations of data (e.g. Box-Cox, or the like) in response to detecting an unusual distribution in numeric features.” [0093] “The quality analysis module 202 may compare a statistical distribution of outcomes from a machine learning model generated by the predictive analytic module 206 to a statistical distribution of initialization data (e.g., training data, testing data, or the like).” [0110] “The model-readiness module 302 may provide an overall model-readiness score, multiple model-readiness scores by category, or the like) for an overall dataset/design matrix, for each feature within a dataset, for one or more label/target/dependent variables, or the like. Based on one or more data quality issues identified by the quality analysis module 202. The model-readiness module 302 may present a “scorecard” or other summary of multiple scores for training data or workload data (e.g., an overall score; sub-scores for different features, different data sets, different identified problem or error types; or the like).”) [Note: The quality analysis module 202, may process data to detect an unusual distribution in numeric features, and how the features fit a common distribution. The process of analyzing the distribution of numeric features and detecting unusual distribution, which is part of assessing the factors contributing to class imbalance. Additionally, the recommendation for transformation of data in response to detecting unusual distribution would correspond to the assessment process and addressing factors relate to the class imbalance. The model-readiness module identifies data quality issues, which would include factors contributing to class imbalance, and the model-readiness module generates readiness scores based on these issues.]
Regarding Previously Presented Claim 6, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
Lorena further teaches: wherein the class overlap attribute indicates an amount of overlap between classes within the dataset. (Lorena, [107: 8] “… where no
(
f
m
i
n
(
T
l
)
)
measures the number of points in the overlapping region of feature
f
m
i
n
for the dataset from the lth round (Tl).” [107: 11] “Higher N1 values indicate the need for more complex boundaries to separate the classes and/or that there is a large amount of overlapping between the classes.” [107: 17] “In complex datasets, in which a high overlapping of the classes is observed, strong vertexes will tend to be less connected to strong neighbors. However, for simple datasets there will be dense regions within the classes and higher hub scores.” Furthermore, see Lorena [Pp. 5, Section: 2.1.3] Volume of Overlapping Region (F2).)
Regarding Previously Presented Claim 8, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
Maughan further teaches: wherein the label purity attribute indicates an accuracy of labels within the dataset. (Maughan, [0094] teaches, “The quality analysis module 202 may perform a statistical analysis of the classes or sets. For example, the quality analysis module 202 may determine that a size and/or ratio of one or more classes or sets has changed and/or drifted over time, or the like. In one embodiment, the quality analysis module 202 may monitor and/or analyze confidence metrics from the machine learning model to detect drift (e.g., if a distribution of confidence metrics becomes bimodal and/or exhibits a different change).”)
Regarding Original Claim 9,
the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
Maughan further teaches: wherein the assessing a quality of the label purity attribute comprises identifying both noisy and confusing data labels within the dataset. (Maughan, [0089] “The quality analysis module 202 may identify and/or quantify drift in a variety of ways. For example, the quality analysis module 202 may take a dataset, a single feature vector, a sample of a dataset (e.g. first 10% captured and last 10% captured or the like), add a binary label based on when the data was captured, and build and score a binary classification model, or the like. If the binary classification model is able to differentiate between “older” and “newer” observations, this is likely caused by drift.,.…, In response to detecting drift, the corrective action module 204 may perform and/or recommend a corrective action such as excluding the features with drift above a threshold, repairing the features with drift by imputing and/or transforming values, or the like.”) [Examiner’s note: labels/outputs that drift would read on noisy and confusing labels.]
Regarding Original Claim 10, the combination of Maughan, Lorena, Devin, Nicholson, Hall, Weston, Tang, and Melvin teaches the elements of claim 1 as outlined above, and further teaches:
Maughan further teaches: wherein the providing an explanation comprises identifying at least one data point within the dataset causing the low quality score. (Maughan, [0072] “The quality analysis module 202, …, may detect one or more unique identifiers in data, and the corrective action module 204 may recommend and/or take one or more associated corrective actions. For example, a unique id feature comprising only unique identifiers may be part of a dataset, but often should not be included in modeling, etc. See paragraph [0072]. Furthermore, Maughan teaches [0076] “the quality analysis module 202 determine that a data quality issue exists the cardinality violates a threshold (e.g., if the cardinality is too high, or too low) the quality analysis module 202 identifying and/or detecting a data quality issue such as an error or other problem or potential problem in data.” See paragraphs [0072] and [0075] - [0076].)
Regarding Amended Claim 11,
The claim recites substantially similar limitation as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. Claim 1 is directed to a method, and claim 11 is directed to an apparatus comprising: at least one processor; and a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor...
Maughan also teaches: Fig. 1, element 102 [Predictive analytics Apparatus], element 104 [Data Sources], and element 106 [Data Network]. [0005] “A computer program product includes a computer readable storage medium storing computer usable program code executable to perform operations. In one embodiment, operations include electronically identifying one or more data quality issues in machine learning training data.” [0013] “Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable storage media having computer readable program code embodied thereon.” [0022] “These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts.”
Regarding Amended Claim 12,
The claim recites substantially similar limitation as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. Claim 1 is directed to a method, and claim 12 is directed to a computer program product, comprising: a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor, ….
Maughan also teaches: [0013] “A computer program product embodied in one or more computer readable storage media having computer readable program code embodied thereon.” [0022] “Aspects of the present disclosure are described below with reference to schematic flowchart diagrams and/or schematic block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the disclosure., …., These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the schematic flowchart diagrams and/or schematic block diagrams block or blocks.”
Regarding Amended Claim 14,
The claim recites substantially similar limitations as corresponding claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Regarding Previously Presented Claim 15,
The claim recites substantially similar limitations as corresponding claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Regarding Original Claim 16,
The claim recites substantially similar limitations as corresponding claim 5 and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Regarding Previously Presented Claim 17,
The claim recites substantially similar limitations as corresponding claim 6 and is rejected for similar reasons as claim 6 using similar teachings and rationale.
Regarding Previously Presented Claim 19,
The claim recites substantially similar limitations as corresponding claim 8 and is rejected for similar reasons as claim 8 using similar teachings and rationale.
Regarding Original Claim 20,
The claim recites substantially similar limitations as corresponding claim 9 and is rejected for similar reasons as claim 9 using similar teachings and rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SADIK ALSHAHARI whose telephone number is (703)756-4749. The examiner can normally be reached Monday Friday, 9 A.M - 6 P.M. ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached on (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.A.A./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights