DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-21 are rejected under 35 U.S.C. 103 as being unpatentable over
SMITH (WO 2024107426 A1), and in view of VINOD (US 20230230352 A1), and further in view of PASUMARTHI (US 20250094619 A1, Date Filed: 2023-09-14).
Re Claim 1, SMITH discloses A method, comprising:
obtaining a knowledge dataset comprising data items, each data item of the knowledge dataset comprising an image, a visual annotation, and a textual annotation (see SMITH: e.g., -- generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models. The disclosed methodology is particularly helpful in generating labels or annotations for unstructured data. In some embodiments, the disclosed approach may be used with data in the form of text, images, or other form of unstructured data.--, in abstract; -- Supervised ML models are trained by applying an ML algorithm to a labeled training dataset. Each data example (or element, in the form of variables, characteristics, or “features”) in the training dataset is associated with a label (or annotation) that defines how the element should be classified by the trained model. A trained model can operate on a previously unseen data example to generate a predicted label as an output. [0003] The performance of an ML model is heavily dependent on the quality and quantity of training data used to produce it. If the model is trained on a training dataset where a significant portion of the data examples are labeled incorrectly (for example, due to human misinterpretation during the annotation process), then the model will learn to "predict" or infer the wrong labels and be of lower accuracy and quality. Conversely, if an ML model is trained on a large enough quantity of high-quality data, it will generalize better when considering previously unseen data points.--, in [0002], and, -- a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models. The disclosed methodology is particularly helpful in generating labels or annotations for unstructured data. In some embodiments, the disclosed approach may be used with data in the form of text, images, or other form of unstructured data. [0014] The disclosed methodology is intended to accelerate the development process for programmatic labeling by automatically identifying and visually representing clusters of salient patterns in data sets, or predictions from language models queried with specific input. Humans with domain knowledge can then review these model outputs and use them to programmatically label data.--, in [0012]-[0015]);
embedding textual annotations of the data items of the knowledge dataset to generate annotation embedding vectors for each of the textual annotations of the data items (see SMITH: e.g., --For an arbitrary dataset, generate a real-valued representation for each datapoint using techniques including one or more of text embeddings, image embeddings, or tf-idf (term frequency–inverse document frequency) vectors, as non-limiting examples, and depending on the type or format of the input data; o Data modalities are turned into a real-valued vector, which is termed an "embedding". The technique to turn a datapoint into an embedding varies depending on the task, data type, and engineering requirements; ^ As an example, for fast text search, tf-idf vectors are sufficient because they are relatively simple to compute compared to generating deep learning embeddings.--, in [0016]);
clustering the annotation embedding vectors of the textual annotations to generate a set of clusters, each cluster associated with a corresponding set of data items of the knowledge data set (see SMITH: e.g., --x Attempt to group (cluster) the datapoints in a dataset using techniques that assign datapoints to the same group if they share similarities. Examples of such assignment algorithms include but are not limited to DBSCAN or distance-based hierarchical clustering. The degree of similarity can be measured by the similarity between two embeddings, or whether two datapoints share the same ground truth labels; o Common similarity metrics are Manhattan distance or Euclidean/Cosine distance, although others exist and may be used. Manhattan distance measures the discrete absolute difference between two quantities, whereas Euclidean distance measures the distance between two points in Euclidean space. Another option is cosine distance, which measures the angle that separates two vectors; o For clustering, Euclidean distance is commonly used to determine whether a datapoint is more likely to belong in one cluster over another by measuring the distance between the datapoint and the centroids of the clusters. To measure the similarity between two datapoints, cosine distance is most commonly used; x Once the datapoints are initially clustered, the process represents each cluster with a unique aggregate or set of attributes, based on (typically) shared attributes of the individual data points in the cluster. These attributes may include but are not limited to shared attributes/aspects of each datapoint that are generally not shared by datapoints outside of the cluster and thus differentiate or distinguish points in a cluster from points not in a cluster--, in [0016]-[0017]);
training a computer vision based machine learning model based on each cluster, wherein the computer vision based machine learning model for a cluster is trained based on images and visual annotations of the set of data items corresponding to that cluster (see SMITH: e.g., -- x For each cluster, the process then trains a classifier to classify datapoints as residing in the cluster or not residing in the cluster. Datapoints that are already in the cluster are included in a positive training dataset to train the classifier. Datapoints that are not in the cluster are included in a negative training dataset; o As a non-limiting example, a SVM (Support Vector Machine) may be used as an algorithm to train a classifier or model; ^ Note that other approaches may be used for the purpose of classifying a "new" datapoint as belonging to or not belonging to a cluster. These other approaches include but are not limited to a centroid + radius approach, or a "bag" of common words (i.e., the use of n-grams as a keyword for a labeling function); o A classifier is developed for each cluster and used to "predict" if a "new" (previously unseen or unclassified) datapoint belongs in that cluster; x The process then stores the classifier for each cluster in a database for future reference. The process associates a classifier with a cluster using a cluster's unique identifier; x For new or previously unclassified datapoints, the process applies the appropriate classifier for each cluster to each of the datapoints. Each classifier generates a "prediction" or likelihood as to whether the datapoint belongs in the associated cluster. These predictions can be leveraged for use cases including, but not limited to, generating programmatic labels for training ML models; o For example, if a new datapoint "belongs" to a particular cluster based on the output of one or more classifiers, then the identifier or an attribute of that cluster can be assigned as a label for that datapoint, and a combination of multiple such labels and datapoints can be used to train a model; o As a non-limiting example, consider the following situation: ^ Assume it is desired to classify a set of emails as spam or not spam. The process flow would first cluster the emails, and for each detected/identified cluster, the process would train a classifier to predict whether a given datapoint belongs in the cluster or not by providing a positive training set as points in the cluster, and a negative training set as the points that are not in the cluster. For this example, assume this results in 10 clusters; ^ Assign each cluster as either HAM or SPAM depending on how many datapoints in each class are in each cluster (this may be based on a majority or threshold value of the assignment of datapoints in a cluster, as an example). One could also ask a user to manually label the clusters for uncertain cases; ^ For data in the dataset that is not labeled, the process would then ask each classifier to predict whether the datapoint is in the cluster or not in the cluster. The threshold value can be set as 0.5 for this task as it is a binary classification problem. Therefore, the process would generate 10 predictions (HAM, SPAM) for each datapoint; ^ The predictions provide a set of weakly supervised labels that may be used downstream to generate annotated training data for a model…automatically generating labels for a set of data used to train a machine learning model.--, in [0016]-[0017]; --the use of a generative model in combination with a discriminative model as part of a process to generate labels for use in training a machine learning model, in accordance with some embodiments; [0023] Figure 1(e) is a flowchart or flow diagram illustrating a method, process, or set of steps, stages, functions, or operations for generating labels or annotations for data used to train a model--, in [0022]-[0023], and, --efficient creation and clustering of embeddings generated from a dataset and use of the formed clusters to programmatically label data. This transforms a large unlabeled and unstructured dataset into labeled training data for use in developing a classifier or other form of machine learning model. [0037] Embodiments of the disclosed approach provide several important benefits. These include the ability to explore and understand data more efficiently (even for cold-start problems), based on insight into semantic clustering of data points using embedding techniques. In addition, embodiments make this insight more actionable with programmatic labeling to intelligently auto- label data at scale (driven in some cases by a user's guidance). Further, training data labeling workflows may be accelerated and efficiently scaled using auto-generated cluster labeling functions which a user can accept and apply with the selection of a user interface element. [0038] In some embodiments, language embedding methods may be used to generate "clusters" of data elements (where the data elements may be words or phrases, field labels, or similar information) that appear to be semantically related. The clusters resulting from a set of training data may vary depending on one or more of the embedding technique used, the metric used to determine similarity for purposes of clustering, or the metric threshold value suggesting that two data elements belong in the same cluster or do not belong in the same cluster (as examples). [0039] Each cluster may be examined by a user and assigned a "label", which is in turn assigned to each data point within the cluster for purposes of training a machine learning model.--, in [0036]-[0039]);
generating a set of classification models/parameters/classifiers based on each cluster, each classification model comprising the computer vision based model trained for an associated cluster (see SMITH: e.g., -- [0036] Embodiments of the disclosed approach enable the efficient creation and clustering of embeddings generated from a dataset and use of the formed clusters to programmatically label data. This transforms a large unlabeled and unstructured dataset into labeled training data for use in developing a classifier or other form of machine learning model. [0037] Embodiments of the disclosed approach provide several important benefits. These include the ability to explore and understand data more efficiently (even for cold-start problems), based on insight into semantic clustering of data points using embedding techniques. In addition, embodiments make this insight more actionable with programmatic labeling to intelligently auto- label data at scale (driven in some cases by a user's guidance). Further, training data labeling workflows may be accelerated and efficiently scaled using auto-generated cluster labeling functions which a user can accept and apply with the selection of a user interface element. [0038] In some embodiments, language embedding methods may be used to generate "clusters" of data elements (where the data elements may be words or phrases, field labels, or similar information) that appear to be semantically related. The clusters resulting from a set of training data may vary depending on one or more of the embedding technique used--, in [0036]-[0038]; and, -- [0059] Programmatic labeling can be applied to many types of supervised learning problems. As examples, it may be applied to text data (long and short), conversations, time series data, PDFs, images, and videos, as well as other forms of data. The disclosed and/or described “labeling function” is flexible enough that the same workflow and framework applies in most cases. As non- limiting examples, potential use cases may include: x Text and/or document classification; x Information extraction from unstructured text, PDF, or HTML; x Rich document processing; x Structured data classification; x Conversational AI and utterance classification; x Entity linking; x Image and cross-modal classification; or x Time series analysis. [0060] Figure 1(e) is a flowchart or flow diagram illustrating a method, process, or set of steps, stages, functions, or operations for generating labels or annotations for data used to train a model, in accordance with some embodiments. As shown in the figure, in some embodiments, the method, process, or set of steps, stages, functions, or operations may include: x Generate Real-Valued Representation for Each Datapoint in a Dataset (as suggested by step or stage 102); o As disclosed, this may involve a technique chosen based on the type of data and/or the task for which a model is to be trained; o In one embodiment, more than a single representation may be generated for some or all of the datapoints in a dataset; ^ Users can review visual representations of the clusters generated by using different representation methods to identify the representation with the most human-understandable clusters… In some embodiments, the unique identifier may be generated by a process that determines one or more common features of the grouped datapoints that distinguish them from the members of the other groups or clusters, such as the presence or absence of a characteristic, the presence or absence of a word or phrase, the presence or absence of an object, or a state of a system or process represented by the datapoint; Typically, attributes are chosen in a way that reflects the uniqueness of a datapoint for a task. As non-limiting examples, this may be performed by: ^ A one-class classification model trained on the attributes of the data points, using the points in the cluster as positive examples--, in [0059]-[0060]);
SMITH however does not explicitly disclose generating a set of classification pipelines,
VINOD discloses generating a set of classification pipelines (see VINOD: e.g., -- [0027] In step 206, based on the ML classifier decision, it executes an ML-based processing pipeline to further process the digital image. The ML-based processing pipeline is defined in a modular way by composing task-specific state-of-the-art ML models for functions such as image normalization, deep learning-based object detection, content extraction, content summarization and image classification. The pipeline may execute in a hybrid fashion, with a mixture of locally executing ML models and cloud-based models.
[0028] For locally installed models, in step 208, model quantization is employed to reduce the model size and compute requirements to fit the resource budget of the smart phone. It is noted that processing operations involving large deep learning models can be offloaded to the cloud. This hybrid pipeline enables optimization on the speed of execution. Based on the execution of the ML pipeline, the mobile application provides a set of contextually relevant choices to the user in step 210.--, in [0027]-[0029]);
SMITH and VINOD are combinable as they are in the same field of endeavor: trained machine learning with labeled texture and images data for prediction, classification based on image data. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify SMITH’s method using VINDO’s teachings by including generating a set of classification pipelines to SMITH’s obtaining/training machine learning models/parameters/classifiers in order to optimization on the speed of execution (see VINOD: e.g., in [0027]-[0029]);
SMITH as modified by VINOD further disclose applying the set of machine learning models and classification pipelines to the image to detection, make prediction, classification (see SMITH: e.g., --[0003] The performance of an ML model is heavily dependent on the quality and quantity of training data used to produce it. If the model is trained on a training dataset where a significant portion of the data examples are labeled incorrectly (for example, due to human misinterpretation during the annotation process), then the model will learn to "predict" or infer the wrong labels and be of lower accuracy and quality. Conversely, if an ML model is trained on a large enough quantity of high-quality data, it will generalize better when considering previously unseen data points.--, in [0002], and, -- a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models. The disclosed methodology is particularly helpful in generating labels or annotations for unstructured data. In some embodiments, the disclosed approach may be used with data in the form of text, images, or other form of unstructured data. [0014] The disclosed methodology is intended to accelerate the development process for programmatic labeling by automatically identifying and visually representing clusters of salient patterns in data sets, or predictions from language models queried with specific input. Humans with domain knowledge can then review these model outputs and use them to programmatically label data.--, in [0012]-[0015]; and see VINOD: e.g., : e.g., -- [0027] In step 206, based on the ML classifier decision, it executes an ML-based processing pipeline to further process the digital image. The ML-based processing pipeline is defined in a modular way by composing task-specific state-of-the-art ML models for functions such as image normalization, deep learning-based object detection, content extraction, content summarization and image classification. The pipeline may execute in a hybrid fashion, with a mixture of locally executing ML models and cloud-based models.
[0028] For locally installed models, in step 208, model quantization is employed to reduce the model size and compute requirements to fit the resource budget of the smart phone. It is noted that processing operations involving large deep learning models can be offloaded to the cloud. This hybrid pipeline enables optimization on the speed of execution. Based on the execution of the ML pipeline, the mobile application provides a set of contextually relevant choices to the user in step 210.--, in [0027]-[0029]);
SMITH as modified by VINOD however still do not explicitly disclose {applying ML models} to make a security violation determination with respect to the image;
PASUMARTHI discloses {applying ML models} to make a security violation determination with respect to the image (see PASUMARTHI: e.g., -- Certain aspects of the disclosure concern a computer-implemented method for improved data security in large language models. The method includes receiving a prompt query entered through a user interface, extracting a plurality of named entities from the prompt query and classifying the plurality of named entities into respective entity classes, tagging the plurality of named entities to be security compliant or security noncompliant based on the respective entity classes, and responsive to finding that one or more named entities are tagged to be security noncompliant, generating an alert on the user interface.--, in abstract, and, -- detecting and preventing data privacy and security violations in chatbot interactions. The solution can identify and warn users when they enter sensitive information that may be exposed or misused by chatbot providers or other chatbot users. As a result, data security of integrating LLMs in enterprise environment can be improved.--, in [0014], and, -- [0060] A user 410 can enter a prompt query 420 via a user interface (e.g., the user interface 122). Upon receiving the prompt query 420, an NER parser 430 (similar to the NER parser 124) can extract and classify, in runtime, a plurality of named entities from the prompt query 420. Each extracted named entity can also be automatically tagged, in runtime, to be security compliant or security noncompliant. The NER parser 430 can perform real-time extraction, classification, and tagging of the named entities based on a custom NER model 480 (like the custom NER model 148) which has been pre-trained based on a text corpus, as described above. Next, a highlighter 440 (like the highlighter 126) can highlight the named entities extracted from the prompt query 420 on the user interface. Different highlighting can be used to distinguish security compliant named entities from security noncompliant named entities. Then, a condition check can be performed at 450. If all extracted named entities are security compliant, the prompt query 420 can be stored in a prompt log 460 (like the prompt log 130) and then submitted as an input to the LLM 470. On the other hand, if at least one of the extracted named entities is security noncompliant, an alert 490 can be generated and fed back to the user 410. The alert 490 can notify the user 410 that the prompt query 420 cannot be submitted to the LLM for potential violation and data privacy and security protocols, and recommend removing the security noncompliant named entities from the prompt query 420. In some implementations, the user 410 can choose to ignore the alert 490. For example, the user 410 can provide annotations 495 (e.g., manually change the tags of some named entities from security noncompliant to security compliant) through the user interface. Then, the prompt query 420 can be saved in the prompt log 460 (along with the user's annotations) and submitted to the LLM 470.--, in [0060]);
SMITH (as modified by VINOD) and PASUMARTHI are combinable as they are in the same field of endeavor: applying the set of machine learning models and classification pipelines to the image to extracting/detection, make prediction, classification. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify SMITH (as modified by VINOD)’s method using PASUMARTHI’s teachings by including {applying ML models} to make a security violation determination with respect to the image to SMITH (as modified by VINOD)’s machine learning models in order to apply LLM for potential violation and data privacy and security protocols (see PASUMARTHI: e.g., in abstract, [0014] and [0060]).
Re Claim 2, SMITH as modified by VINOD and PASUMARTHI further disclose training a semantic segmentation model associated with each of one or more clusters, wherein the semantic segmentation model for the cluster is trained based on images and visual annotations of the set of data items corresponding to that cluster (see SMITH: e.g., -- [0012] In the context of this disclosure, a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models.--, in [0012]; -- [0037] Embodiments of the disclosed approach provide several important benefits. These include the ability to explore and understand data more efficiently (even for cold-start problems), based on insight into semantic clustering of data points using embedding techniques. In addition, embodiments make this insight more actionable with programmatic labeling to intelligently auto- label data at scale (driven in some cases by a user's guidance). Further, training data labeling workflows may be accelerated and efficiently scaled using auto-generated cluster labeling functions which a user can accept and apply with the selection of a user interface element. [0038] In some embodiments, language embedding methods may be used to generate "clusters" of data elements (where the data elements may be words or phrases, field labels, or similar information) that appear to be semantically related. The clusters resulting from a set of training data may vary depending on one or more of the embedding technique used, the metric used to determine similarity for purposes of clustering, or the metric threshold value suggesting that two data elements belong in the same cluster or do not belong in the same cluster (as examples).--, in [0037]-[0038]; and,
-- [0078] The auto-generated labeling functions provide a mechanism to bootstrap a labeling effort, and the insights from cluster exploration may provide inspiration for additional labeling functions that are useful for the dataset or for a different dataset. [0079] In some embodiments, the disclosed and/or described processing flow takes a large, unstructured dataset of complex text (or other type of) documents and provides a visualization of embedding-based clustering. A user can inspect each cluster to understand the meaning behind it and explore explicit data points. A user can filter the proposed clusters using the search functionality to see how specific "slices" or segments of data distribute across clusters and uncover additional nuance to the dataset. [0080] As a user explores and understands the proposed clusters, they can take informed actions by saving and applying auto-generated labeling functions that are used to programmatically label a dataset. This can be followed by continuing with the core functionality of the overall workflow to label data, generate a trained model, and adapt.--, in [0078]-[0080]); and
wherein the classification pipelines associated with each of the one or more clusters comprises the semantic segmentation model associated that cluster (see VINOD: e.g., -- [0027] In step 206, based on the ML classifier decision, it executes an ML-based processing pipeline to further process the digital image. The ML-based processing pipeline is defined in a modular way by composing task-specific state-of-the-art ML models for functions such as image normalization, deep learning-based object detection, content extraction, content summarization and image classification. The pipeline may execute in a hybrid fashion, with a mixture of locally executing ML models and cloud-based models.
[0028] For locally installed models, in step 208, model quantization is employed to reduce the model size and compute requirements to fit the resource budget of the smart phone. It is noted that processing operations involving large deep learning models can be offloaded to the cloud. This hybrid pipeline enables optimization on the speed of execution. Based on the execution of the ML pipeline, the mobile application provides a set of contextually relevant choices to the user in step 210.--, in [0027]-[0029]).
Re Claim 3, SMITH as modified by VINOD and PASUMARTHI further disclose wherein the visual annotation is a bounding shape (see PASUMARTHI: e.g., -- [0063] As shown in FIG. 7, all extracted named entities are highlighted (e.g., with grey text background) and their corresponding entity classes are displayed next to the respective named entities. In this example, the security noncompliant named entities are further bounded within rectangular text boxes so that they are visually distinguished from the security compliant named entities. In some implementations, the user can right click a named entity to manually change the tagging result of a named entity (e.g., switching from security noncompliant to security compliant) and add annotations (e.g., providing reasons for the change.--, in [0063]).
Re Claim 4, SMITH as modified by VINOD and PASUMARTHI further disclose wherein the textual annotation is in natural language (see SMITH: e.g., -- By learning a generative model, and directly estimating P(L|y), the process is essentially learning the relative accuracies of the labeling functions based on how they overlap and conflict. [0054] Embodiments then use this estimated generative model over the labeling functions to train a noise-aware version of an end discriminative model. To do so, the generative model infers probabilities over the unknown labels of the training data, and then the process minimizes the expected loss of the discriminative model with respect to these probabilities. [0055] Estimating the parameters of a generative model can be complicated, especially when there are statistical dependencies between the labeling functions used (either user-expressed or inferred). Work performed by the inventors suggests that given sufficient labeling functions, one can obtain similar asymptotic scaling as with supervised methods in many use cases of interest. The inventors also investigated how the process can learn correlations among the labeling functions without using labeled data and how that can improve performance of a model training process. [0056] A weak supervision interaction model (parts of which are disclosed and/or described herein, and further disclosed and/or described in U.S. Patent Application Serial Number 18/214,024) may be extended to other modalities or tasks, including richly formatted data and images, supervising tasks with natural language, and generating labeling functions automatically. Extending the core data programming model is expected to make it possible to specify labeling functions with higher-level interfaces such as natural language, as well as assist in combining labeling functions with other types of weak supervision, such as data augmentation.--, in [0053]-[0056]).
Re Claim 5, SMITH as modified by VINOD and PASUMARTHI further disclose determining an attack vector descriptor associated with each classification pipeline based on each cluster, wherein the attack vector descriptor for the cluster is generated based on the textual annotations of the set of data items corresponding to that cluster and expresses a concept associated those textual annotations (see Smith: e.g., -- In some cases, an ID or attribute may be a randomly generated string of numbers/characters. For example, if two documents in a dataset are different, one might choose to represent them with two different unique identifiers. If two images were the same, one would represent them with the same unique identifier; As described, in some embodiments, the process flow generates a summary descriptor(s) for a cluster so that a human can glance at it and better understand its contents. This is useful because a human may use this descriptor or set of attributes to determine a label; ^ In one implementation of an embodiment, the descriptor is the set of the most “characteristic” unigrams/bigrams/trigrams in a cluster as compared to others; ^ One could generate this type of descriptor by using a TF-IDF matrix, where a process concatenates the contents of each data point in a cluster into a single document. One could also pass several of the documents from the cluster to a large language model (LLM) and ask it to summarize the contents into a set of keywords or to generate a possible label; x For Each Group or Cluster, Train a Classifier to Classify a Datapoint as Either Inside or Outside the Cluster (step or stage 108); o This will result in a set of classifiers, with one corresponding to each of the formed groups or clusters; o Each such classifier may be evaluated using a set of datapoints to determine the classifier's accuracy and the utility of the assigned identifier (which may later serve as a label for datapoints assigned to the cluster);--, in [0060]).
Re Claim 6, SMITH as modified by VINOD and PASUMARTHI further disclose wherein the attack vector descriptors are generated by providing a prompt and the textual annotations to a large language model (see SMITH: e.g., -- [0012] In the context of this disclosure, a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models.--, in [0012]-[0013]; and also see PASUMARTHI: e.g., -- Certain aspects of the disclosure concern a computer-implemented method for improved data security in large language models. The method includes receiving a prompt query entered through a user interface, extracting a plurality of named entities from the prompt query and classifying the plurality of named entities into respective entity classes, tagging the plurality of named entities to be security compliant or security noncompliant based on the respective entity classes, and responsive to finding that one or more named entities are tagged to be security noncompliant, generating an alert on the user interface.--, in abstract, and, -- detecting and preventing data privacy and security violations in chatbot interactions. The solution can identify and warn users when they enter sensitive information that may be exposed or misused by chatbot providers or other chatbot users. As a result, data security of integrating LLMs in enterprise environment can be improved.--, in [0014], and, -- [0060] A user 410 can enter a prompt query 420 via a user interface (e.g., the user interface 122). Upon receiving the prompt query 420, an NER parser 430 (similar to the NER parser 124) can extract and classify, in runtime, a plurality of named entities from the prompt query 420. Each extracted named entity can also be automatically tagged, in runtime, to be security compliant or security noncompliant. The NER parser 430 can perform real-time extraction, classification, and tagging of the named entities based on a custom NER model 480 (like the custom NER model 148) which has been pre-trained based on a text corpus, as described above. Next, a highlighter 440 (like the highlighter 126) can highlight the named entities extracted from the prompt query 420 on the user interface. Different highlighting can be used to distinguish security compliant named entities from security noncompliant named entities. Then, a condition check can be performed at 450. If all extracted named entities are security compliant, the prompt query 420 can be stored in a prompt log 460 (like the prompt log 130) and then submitted as an input to the LLM 470. On the other hand, if at least one of the extracted named entities is security noncompliant, an alert 490 can be generated and fed back to the user 410. The alert 490 can notify the user 410 that the prompt query 420 cannot be submitted to the LLM for potential violation and data privacy and security protocols, and recommend removing the security noncompliant named entities from the prompt query 420. In some implementations, the user 410 can choose to ignore the alert 490. For example, the user 410 can provide annotations 495 (e.g., manually change the tags of some named entities from security noncompliant to security compliant) through the user interface. Then, the prompt query 420 can be saved in the prompt log 460 (along with the user's annotations) and submitted to the LLM 470.--, in [0060]).
Re Claim 7, SMITH as modified by VINOD and PASUMARTHI further disclose receiving a new data item, wherein the new data item is associated with a first attack vector descriptor (see SMITH: e.g., -- [0012] In the context of this disclosure, a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models.--, in [0012]-[0013]; and also see PASUMARTHI: e.g., -- Certain aspects of the disclosure concern a computer-implemented method for improved data security in large language models. The method includes receiving a prompt query entered through a user interface, extracting a plurality of named entities from the prompt query and classifying the plurality of named entities into respective entity classes, tagging the plurality of named entities to be security compliant or security noncompliant based on the respective entity classes, and responsive to finding that one or more named entities are tagged to be security noncompliant, generating an alert on the user interface.--, in abstract, and, -- detecting and preventing data privacy and security violations in chatbot interactions. The solution can identify and warn users when they enter sensitive information that may be exposed or misused by chatbot providers or other chatbot users. As a result, data security of integrating LLMs in enterprise environment can be improved.--, in [0014], and, -- [0060] A user 410 can enter a prompt query 420 via a user interface (e.g., the user interface 122). Upon receiving the prompt query 420, an NER parser 430 (similar to the NER parser 124) can extract and classify, in runtime, a plurality of named entities from the prompt query 420. Each extracted named entity can also be automatically tagged, in runtime, to be security compliant or security noncompliant. The NER parser 430 can perform real-time extraction, classification, and tagging of the named entities based on a custom NER model 480 (like the custom NER model 148) which has been pre-trained based on a text corpus, as described above. Next, a highlighter 440 (like the highlighter 126) can highlight the named entities extracted from the prompt query 420 on the user interface. Different highlighting can be used to distinguish security compliant named entities from security noncompliant named entities. Then, a condition check can be performed at 450. If all extracted named entities are security compliant, the prompt query 420 can be stored in a prompt log 460 (like the prompt log 130) and then submitted as an input to the LLM 470. On the other hand, if at least one of the extracted named entities is security noncompliant, an alert 490 can be generated and fed back to the user 410. The alert 490 can notify the user 410 that the prompt query 420 cannot be submitted to the LLM for potential violation and data privacy and security protocols, and recommend removing the security noncompliant named entities from the prompt query 420. In some implementations, the user 410 can choose to ignore the alert 490. For example, the user 410 can provide annotations 495 (e.g., manually change the tags of some named entities from security noncompliant to security compliant) through the user interface. Then, the prompt query 420 can be saved in the prompt log 460 (along with the user's annotations) and submitted to the LLM 470.--, in [0060]);
determining the classification pipeline associated with the new data item based on the first attack vector descriptor (see SMITH: e.g., -- [0012] In the context of this disclosure, a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models.--, in [0012]-[0013]; and, see VINOD: e.g., -- [0027] In step 206, based on the ML classifier decision, it executes an ML-based processing pipeline to further process the digital image. The ML-based processing pipeline is defined in a modular way by composing task-specific state-of-the-art ML models for functions such as image normalization, deep learning-based object detection, content extraction, content summarization and image classification. The pipeline may execute in a hybrid fashion, with a mixture of locally executing ML models and cloud-based models.
[0028] For locally installed models, in step 208, model quantization is employed to reduce the model size and compute requirements to fit the resource budget of the smart phone. It is noted that processing operations involving large deep learning models can be offloaded to the cloud. This hybrid pipeline enables optimization on the speed of execution. Based on the execution of the ML pipeline, the mobile application provides a set of contextually relevant choices to the user in step 210.--, in [0027]-[0029];
also see PASUMARTHI: e.g., -- Certain aspects of the disclosure concern a computer-implemented method for improved data security in large language models. The method includes receiving a prompt query entered through a user interface, extracting a plurality of named entities from the prompt query and classifying the plurality of named entities into respective entity classes, tagging the plurality of named entities to be security compliant or security noncompliant based on the respective entity classes, and responsive to finding that one or more named entities are tagged to be security noncompliant, generating an alert on the user interface.--, in abstract, and, -- detecting and preventing data privacy and security violations in chatbot interactions. The solution can identify and warn users when they enter sensitive information that may be exposed or misused by chatbot providers or other chatbot users. As a result, data security of integrating LLMs in enterprise environment can be improved.--, in [0014], and, -- [0060] A user 410 can enter a prompt query 420 via a user interface (e.g., the user interface 122). Upon receiving the prompt query 420, an NER parser 430 (similar to the NER parser 124) can extract and classify, in runtime, a plurality of named entities from the prompt query 420. Each extracted named entity can also be automatically tagged, in runtime, to be security compliant or security noncompliant. The NER parser 430 can perform real-time extraction, classification, and tagging of the named entities based on a custom NER model 480 (like the custom NER model 148) which has been pre-trained based on a text corpus, as described above. Next, a highlighter 440 (like the highlighter 126) can highlight the named entities extracted from the prompt query 420 on the user interface. Different highlighting can be used to distinguish security compliant named entities from security noncompliant named entities. Then, a condition check can be performed at 450. If all extracted named entities are security compliant, the prompt query 420 can be stored in a prompt log 460 (like the prompt log 130) and then submitted as an input to the LLM 470. On the other hand, if at least one of the extracted named entities is security noncompliant, an alert 490 can be generated and fed back to the user 410. The alert 490 can notify the user 410 that the prompt query 420 cannot be submitted to the LLM for potential violation and data privacy and security protocols, and recommend removing the security noncompliant named entities from the prompt query 420. In some implementations, the user 410 can choose to ignore the alert 490. For example, the user 410 can provide annotations 495 (e.g., manually change the tags of some named entities from security noncompliant to security compliant) through the user interface. Then, the prompt query 420 can be saved in the prompt log 460 (along with the user's annotations) and submitted to the LLM 470.--, in [0060]); and
retraining only the computer vision based machine learning model of the classification pipeline associated with the new data item, wherein the computer vision based machine learning model is retrained based on the new data item (see SMITH: e.g., -- [0012] In the context of this disclosure, a classifier is a model or algorithm that is used to segment input data into a category, such as by indicating the likelihood of the presence or absence of some characteristic in the data (where as examples, the data may be text or an image). A classifier may be used to assign an identifying label to a set of input data, where the label may represent a class, category, or characteristic of the data. Classifiers may be used to determine an expected or “predicted” output based on a set of input data. Classifiers may be used in the processing of data sets and may be implemented in the form of trained machine learning (ML) models, deep learning (DL) models, or neural networks. Training requires a set of data items and an associated label or annotation for each data item. [0013] Embodiments of the disclosed systems, apparatuses, and methods introduce an approach to semi-automatically (that is, programmatically) generate labels for data based on implementation of a clustering or language model prompting technique and can be used to implement a form of programmatic labeling to accelerate the development of classifiers and other forms of models.--, in [0012]-[0013]; and, see VINOD: e.g., -- [0027] In step 206, based on the ML classifier decision, it executes an ML-based processing pipeline to further process the digital image. The ML-based processing pipeline is defined in a modular way by composing task-specific state-of-the-art ML models for functions such as image normalization, deep learning-based object detection, content extraction, content summarization and image classification. The pipeline may execute in a hybrid fashion, with a mixture of locally executing ML models and cloud-based models.
[0028] For locally installed models, in step 208, model quantization is employed to reduce the model size and compute requirements to fit the resource budget of the smart phone. It is noted that processing operations involving large deep learning models can be offloaded to the cloud. This hybrid pipeline enables optimization on the speed of execution. Based on the execution of the ML pipeline, the mobile application provides a set of contextually relevant choices to the user in step 210.--, in [0027]-[0029];
also see PASUMARTHI: e.g., -- Certain aspects of the disclosure concern a computer-implemented method for improved data security in large language models. The method includes receiving a prompt query entered through a user interface, extracting a plurality of named entities from the prompt query and classifying the plurality of named entities into respective entity classes, tagging the plurality of named entities to be security compliant or security noncompliant based on the respective entity classes, and responsive to finding that one or more named entities are tagged to be security noncompliant, generating an alert on the user interface.--, in abstract, and, -- detecting and preventing data privacy and security violations in chatbot interactions. The solution can identify and warn users when they enter sensitive information that may be exposed or misused by chatbot providers or other chatbot users. As a result, data security of integrating LLMs in enterprise environment can be improved.--, in [0014], and, -- [0060] A user 410 can enter a prompt query 420 via a user interface (e.g., the user interface 122). Upon receiving the prompt query 420, an NER parser 430 (similar to the NER parser 124) can extract and classify, in runtime, a plurality of named entities from the prompt query 420. Each extracted named entity can also be automatically tagged, in runtime, to be security compliant or security noncompliant. The NER parser 430 can perform real-time extraction, classification, and tagging of the named entities based on a custom NER model 480 (like the custom NER model 148) which has been pre-trained based on a text corpus, as described above. Next, a highlighter 440 (like the highlighter 126) can highlight the named entities extracted from the prompt query 420 on the user interface. Different highlighting can be used to distinguish security compliant named entities from security noncompliant named entities. Then, a condition check can be performed at 450. If all extracted named entities are security compliant, the prompt query 420 can be stored in a prompt log 460 (like the prompt log 130) and then submitted as an input to the LLM 470. On the other hand, if at least one of the extracted named entities is security noncompliant, an alert 490 can be generated and fed back to the user 410. The alert 490 can notify the user 410 that the prompt query 420 cannot be submitted to the LLM for potential violation and data privacy and security protocols, and recommend removing the security noncompliant named entities from the prompt query 420. In some implementations, the user 410 can choose to ignore the alert 490. For example, the user 410 can provide annotations 495 (e.g., manually change the tags of some named entities from security noncompliant to security compliant) through the user interface. Then, the prompt query 420 can be saved in the prompt log 460 (along with the user's annotations) and submitted to the LLM 470.--, in [0060]).
Re Claims 8-14, claims 8-14 are corresponding system claim to claims 1-7, respectively. Claims 8-14 thus are rejected for the similar reasons for claims 1-7. See above discussions with regard to claims 1-7 respectively. Furthermore, SMITH as modified by VINOD and PASUMARTHI further disclose A system, comprising: a processor; a data store comprising a knowledge dataset comprising data items, each data item of the knowledge dataset comprising an image, a visual annotation, and a textual annotation; a non-transitory computer readable medium, comprising instructions for performing the method (see SMITH: e.g., Fig. 4, and, -- [0088] In general, an embodiment may be implemented using a set of software instructions that are executed by a suitably programmed processing element (such as a GPU, CPU, TPU, QPU, microprocessor, processor, controller, state machine, or computing device, as non-limiting examples). In a complex application or system such instructions are typically arranged into “modules” with each such module typically performing a specific task, process, function, or operation. The entire set of modules may be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform. [0089] Each application module or sub-module may correspond to a particular function, method, process, or operation that is implemented by the module or sub-module. Such function, method, process, or operation may include those used to implement one or more aspects of the disclosed and/or described systems, apparatuses, and methods. [0090] The modules and/or sub-modules may include a suitable computer-executable code or set of instructions, such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer- executable code. Alternatively, or in addition, the programming language may be an interpreted programming language such as a scripting language. [0091] A module may contain instructions that are executed by a processor contained in more than one of a server, client device, network element, system, platform, or other component. In some embodiments, a plurality of electronic processors, with each being part of a separate device, server, platform, or system may be responsible for executing all or a portion of the software instructions contained in an illustrated module--, in [0088]-[0090]).
Re Claims 15-21, claims 15-21 are corresponding medium claim to claims 1-7, respectively. Claims 15-21 thus are rejected for the similar reasons for claims 1-7. See above discussions with regard to claims 1-7 respectively. Furthermore, SMITH as modified by VINOD and PASUMARTHI further disclose A non-transitory computer readable medium, comprising instructions for performing the method (see SMITH: e.g., Fig. 4, and, -- [0088] In general, an embodiment may be implemented using a set of software instructions that are executed by a suitably programmed processing element (such as a GPU, CPU, TPU, QPU, microprocessor, processor, controller, state machine, or computing device, as non-limiting examples). In a complex application or system such instructions are typically arranged into “modules” with each such module typically performing a specific task, process, function, or operation. The entire set of modules may be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform. [0089] Each application module or sub-module may correspond to a particular function, method, process, or operation that is implemented by the module or sub-module. Such function, method, process, or operation may include those used to implement one or more aspects of the disclosed and/or described systems, apparatuses, and methods. [0090] The modules and/or sub-modules may include a suitable computer-executable code or set of instructions, such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer- executable code. Alternatively, or in addition, the programming language may be an interpreted programming language such as a scripting language. [0091] A module may contain instructions that are executed by a processor contained in more than one of a server, client device, network element, system, platform, or other component. In some embodiments, a plurality of electronic processors, with each being part of a separate device, server, platform, or system may be responsible for executing all or a portion of the software instructions contained in an illustrated module--, in [0088]-[0090]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights