DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 35-37 and 43-46, are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejection Notes
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 21 and 22, are rejected under 35 U.S.C. 102a1 as being anticipated by Oliver et al. (US 20230070616 A1, filed: 9/7/2022).
Claim 21: Oliver teaches an apparatus for verification and analysis of artificial intelligence (Al) models, the apparatus comprising:
a display device having a display screen (a dashboard interface 300 displays [Oliver, 0057]) that is configured to display a first user interface (UI) including: a scatter plot window showing an aggregate view of a plurality of data points associated with images forming a scatter plot (a dashboard interface 300 displays the analysis in various visualizations, including heatmap 302, ranked list 304, and scatter plot 306 [Oliver, 0057]); and
a plurality of statistical charts providing information about prediction performance of an Al model over the images (AI examines vast amounts of data and find the trends and patterns, develop forecasts and analyze potential scenarios, streamline data analysis by funneling all data into one solution. AI may use Tableau, Qlik Sense, Sisense, Power BI, SAS BI, or Google Data Studio. A dashboard interface 300 displays the analysis in various visualizations, including heatmap 302, ranked list 304, and scatter plot 306 [Oliver, 0057]. The AI module or process may also adjust the predictive weights of the variables without human supervision, adjust the threshold values of specific variables without human supervision, or evaluate new variables present in the data feed but not presently used in the predictive model. The AI module or process may compare the actual observed outcome of the event to the predicted outcome, then separately analyze the variables within the model that contributed to the incorrect outcome [Oliver, 0060]).
Claim 22: Oliver teaches the apparatus of claim 21. Oliver further teaches wherein the first user interface further includes: a data view window showing a plurality of images each having one or more associated machine learning labels based on a selection of one or more data points in the scatter plot or a selection of data in a statistical chart (the interactive dashboard provided by the system and method can also be incorporated into other use cases such as predictive models for health services utilization, neighborhood health quality index, and the impact of relaxing public health protections [Oliver, 0051]. A dashboard interface 300 displays the analysis in various visualizations, including heatmap 302, ranked list 304, and scatter plot 306 [Oliver, 0057]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 23 and 25, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Kurpiewski et al. (US 20220324060 A1, published: 10/13/2022).
Claim 23: Oliver teaches the apparatus of claim 22. Oliver does not teach wherein: the Al model is of a classification type and the plurality of statistical charts includes one or more of the group comprising: an interactive bar chart of histograms illustrating prediction confidences; an interactive line plot of precision versus recall; and an interactive matrix plot for a confusion matrix.
However, Kurpiewski teaches wherein: the Al model is of a classification type and the plurality of statistical charts includes one or more of the group comprising: an interactive bar chart of histograms illustrating prediction confidences; an interactive line plot of precision versus recall; and an interactive matrix plot for a confusion matrix (the quality monitoring systems and methods of the present disclosure utilize data analytics and machine learning, including regression, classification, and/or AI models and algorithms to predict or estimate the weld quality for a given weld based on weld parameters that are determined, generated, and/or sensed during the weld process [Kurpiewski, 0051]. A method referred to as Matrix Profiling Anomalies was used to search weld graph data for weld anomalies [Kurpiewski, 0107]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver to include the graphing optional feature of Kurpiewski.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim 25: Oliver teaches the apparatus of claim 22. Oliver does not teach wherein: the AI model is of object detection type and the plurality of statistical charts includes one or more of the group comprising: an interactive matrix plot illustrating prediction confidences versus intersection over union (IoU) scores; an interactive line plot of precision versus recall; and an interactive matrix plot for a confusion matrix.
However, Kurpiewski teaches wherein: the AI model is of object detection type and the plurality of statistical charts includes one or more of the group comprising: an interactive matrix plot illustrating prediction confidences versus intersection over union (IoU) scores; an interactive line plot of precision versus recall; and an interactive matrix plot for a confusion matrix ([Kurpiewski, 0051]; [Kurpiewski, 0107]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver to include the graphing optional feature of Kurpiewski.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 24 and 26, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022) and Kurpiewski et al. (US 20220324060 A1, published: 10/13/2022), and in further view of Lou et al. (US 20200069973 A1, published: 3/5/2020).
Claim 24: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 23. The combination of Oliver and Kurpiewski, does not teach wherein: the first user interface (UI) further displays a slider menu to select a lower prediction confidence threshold for a range of prediction confidence thresholds from the lower prediction confidence threshold to a maximum of one; and a selection of the lower prediction confidence threshold with the slider menu results in changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to the images whose prediction confidence falls within the selected range.
However, Lou teaches wherein: the first user interface (UI) further displays a slider menu to select a lower prediction confidence threshold for a range of prediction confidence thresholds from the lower prediction confidence threshold to a maximum of one; and a selection of the lower prediction confidence threshold with the slider menu results in changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to the images whose prediction confidence falls within the selected range (a user input device (e.g., keyboard, buttons, sliders, dials, trackball, mouse, or other device) is provided for user interaction with the outcome prediction [Lou, 0161]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing control feature of Lou.
One would have been motivated to make this modification to include typical graphing command elements that enable users to select the functions of a graphing application that are useful for completing their tasks.
Claim 26: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 25. The combination of Oliver and Kurpiewski, does not teach wherein: the first user interface (UI) further displays a first slider menu to select a lower prediction confidence threshold value for a first range of prediction confidence thresholds and a second slider menu to select a lower IoU score threshold value for a second range of IoU score thresholds; and wherein a selection of the first slider menu results in changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to images whose prediction confidence falls within the first range; wherein a selection of the second slider menu results in further changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to images whose IoU score threshold falls within the second range.
However, Lou teaches wherein: the first user interface (UI) further displays a first slider menu to select a lower prediction confidence threshold value for a first range of prediction confidence thresholds and a second slider menu to select a lower IoU score threshold value for a second range of IoU score thresholds; and wherein a selection of the first slider menu results in changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to images whose prediction confidence falls within the first range; wherein a selection of the second slider menu results in further changes to the interactive line plot of precision versus recall and the interactive matrix plot for the confusion matrix to correspond to images whose IoU score threshold falls within the second range (a user input device (e.g., keyboard, buttons, sliders, dials, trackball, mouse, or other device) is provided for user interaction with the outcome prediction [Lou, 0161]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing control feature of Lou.
One would have been motivated to make this modification to include typical graphing command elements that enable users to select the functions of a graphing application that are useful for completing their tasks.
Claim(s) 27 and 29, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Lester et al. (US 20220284148 A1, published: 9/8/2022).
Claim 27: Oliver teaches the apparatus of claim 21. Oliver does not teach wherein: the plurality of data points are shown clustering together in clusters based on distances between image features.
However, Lester teaches wherein: the plurality of data points are shown clustering together in clusters based on distances between image features (the parameters may include one or more of a fixed number of clusters, a distance between a data point at a center of a cluster and other data points in the cluster, or a minimum number of data points in a cluster [Lester, 0047]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Lester.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 28, is rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022) and Lester et al. (US 20220284148 A1, published: 9/8/2022), and in further view of Kamkar et al. (US 20220164877 A1, published: 5/26/2022).
Claim 28: The combination of Oliver and Lester, teaches the apparatus of claim 27. The combination of Oliver and Lester, does not teach wherein: the image features are determined by a class-wise evaluation of a divergence between ground truth class-labels and model prediction confidences in the case of image classification.
However, Kamkar teaches wherein: the image features are determined by a class-wise evaluation of a divergence between ground truth class-labels and model prediction confidences in the case of image classification (a data object that represents the ground truth labels for each training data row prediction (e.g., a Y label object class) includes the ground truth labels (Z.sub.label) for the sensitive attributes [Kamkar, 0068]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Lester, to include the graphing data feature of Kamkar.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 29, is rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022) and Lester et al. (US 20220284148 A1, published: 9/8/2022), and in further view of Harrison et al. (US 20200082923 A1, published: 3/12/2020).
Claim 29: The combination of Oliver and Lester, teaches the apparatus of claim 27. The combination of Oliver and Lester, does not teach wherein: the image features are determined by a content of the images.
However, Kurpiewski further teaches wherein: the image features are determined by a content of the images (the quality monitoring system can receive weld parameter data from sensors that are external to the welding process, such as temperature data from temperature sensors, laser vibrometer position measurements, stress/strain data, image data of the resulting weld from a camera, vibration data from an impact sensor indicating the vibration of an ultrasonic stack, etc. [Kurpiewski, 0034]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Lester, to include the graphing optional feature of Kurpiewski.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 30, is rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Harrison et al. (US 20200082923 A1, published: 3/12/2020).
Claim 30: Oliver teaches the apparatus of claim 22. Oliver does not teach wherein: the one or more associated machine learning labels includes at least one classification label.
However, Harrison teaches wherein: the one or more associated machine learning labels includes at least one classification label (the feature extraction circuit 203 extracts the features from the label where the label information extracts features (such as name, medication, dosage, date pharmacy, etc.) Artificial Intelligence (AI) or Machine Language (ML) classifies the information using label classification neural networks 205 [Harrison, 0042]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Harrison.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 31-32, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Laxman et al. (US 20210142291 A1, published: 5/13/2021).
Claim 31: Oliver teaches the apparatus of claim 22. Oliver does not teach wherein: the one or more associated machine learning labels includes at least one object class and location label.
However, Laxman teaches wherein: the one or more associated machine learning labels includes at least one object class and location label (AI-based business assistant 2300 can assume DAG frames with class labels as input [Laxman, 0094]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Laxman.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim 32: Oliver teaches the apparatus of claim 22. Oliver does not teach wherein: the one or more associated machine learning labels includes a ground truth label, a predicted label, or both a ground truth label and a predicted label.
However, Laxman teaches wherein: the one or more associated machine learning labels includes a ground truth label, a predicted label, or both a ground truth label and a predicted label (AI-based business assistant 2300 receives predicted class labels with scores 2214 [Laxman, 0100]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Laxman.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 33, 34, and 42, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Barsoum et al. (US 20130212508 A1, published: 8/15/2013).
Claim 33: Oliver teaches the apparatus of claim 21. Oliver does not teach wherein: the scatter plot window further includes: an inset window including a legend with interactive buttons to control the scatter plot of the data points.
However, Barsoum teaches wherein: the scatter plot window further includes: an inset window including a legend with interactive buttons to control the scatter plot of the data points (the GUI 56 also includes a plurality of GUI elements 60 that can be activated to access additional functions and tools, such as including a "refresh" button, a problem update ("PBL update") button, a "scatter chart" button and a "motion chart" button [Barsoum, 0046]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Barsoum.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim 34: The combination of Oliver, and Barsoum, teaches the apparatus of claim 21. Barsoum further teaches wherein: the scatter plot window further includes: a plurality of option buttons to filter, sample, and view images corresponding to the data points in the scatter plot ([Barsoum, FIGures]; Examiner's Note: as illustrated).
Claim 42: Oliver teaches the apparatus of claim 21. Oliver does not teach further comprising: the display device having a display screen that is configured to display a second user interface window including: a collapsible sidebar with buttons to navigate to different user interface windows to view and add datasets, and to view and add model analysis jobs.
However, Barsoum further teaches further comprising: the display device having a display screen that is configured to display a second user interface window including: a collapsible sidebar with buttons to navigate to different user interface windows to view and add datasets, and to view and add model analysis jobs ([Barsoum, FIGs. 2-16]; Examiner's Note: as illustrated).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of Oliver, to include the graphing data feature of Barsoum.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 38 is rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Zhang et al. (US 20230100537 A1, published: 3/30/2023).
Claim 38: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 23. The combination of Oliver and Kurpiewski, does not teach wherein: the selection of data in a statistical chart comprises: the selection of a bar in an interactive bar chart of histograms illustrating prediction confidences.
However, Zhang teaches wherein: the selection of data in a statistical chart comprises: the selection of a bar in an interactive bar chart of histograms illustrating prediction confidences (in Section 701 of FIG. 7, a user can select a bar of the histogram and then press a key associated with a target category [Zhang, 0088]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing data feature of Zhang.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim(s) 39-41, are rejected under 35 U.S.C. 103 as being unpatentable over Oliver et al. (US 20230070616 A1, filed: 9/7/2022), in view of Sahouria et al. (US 20060236299 A1, published: 10/19/2006).
Claim 39: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 23. The combination of Oliver and Kurpiewski, does not teach wherein: the selection of data in a statistical chart comprises: the selection of a row, a column, a diagonal, or an individual cell in an interactive matrix plot for the confusion matrix.
However, Sahouria teaches wherein: the selection of data in a statistical chart comprises: the selection of a row, a column, a diagonal, or an individual cell in an interactive matrix plot for the confusion matrix (FIG. 3B illustrates a graph of the original IC layout data with cells A, B, C, and I selected as acceptable cover cells [Sahouria ,0037]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing data feature of Sahouria.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim 40: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 25. The combination of Oliver and Kurpiewski, does not teach wherein: the selection of data in a statistical chart comprises: the selection of a cell in an interactive matrix plot illustrating prediction confidences versus intersection over union (IoU) scores.
However, Sahouria teaches wherein: the selection of data in a statistical chart comprises: the selection of a cell in an interactive matrix plot illustrating prediction confidences versus intersection over union (IoU) scores (FIG. 3B illustrates a graph of the original IC layout data with cells A, B, C, and I selected as acceptable cover cells [Sahouria ,0037]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing data feature of Sahouria.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Claim 41: The combination of Oliver and Kurpiewski, teaches the apparatus of claim 25. The combination of Oliver and Kurpiewski, does not teach wherein: the selection of data in a statistical chart comprises: the selection of a row, a column, a diagonal, or an individual cell in an interactive matrix plot for the confusion matrix.
However, Sahouria teaches wherein: the selection of data in a statistical chart comprises: the selection of a row, a column, a diagonal, or an individual cell in an interactive matrix plot for the confusion matrix (FIG. 3B illustrates a graph of the original IC layout data with cells A, B, C, and I selected as acceptable cover cells [Sahouria ,0037]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the invention was filed, to modify the AI prediction scatter graphing invention of the combination of Oliver and Kurpiewski, to include the graphing data feature of Sahouria.
One would have been motivated to make this modification to include typical graphing program features that enable users to plot representations of the data that they are working with.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SETH A SILVERMAN whose telephone number is (571)272-9783. The examiner can normally be reached Mon-Thur, 8AM-4PM MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571)272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Seth A Silverman/Primary Examiner, Art Unit 2172