DETAILED ACTION
This action is in response to the original filing on 12/04/2023. Claims 1-15 are pending examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Claims 1, 8, and 15
Step 1: Claims 1, 8, and 15 recites a method, a system, and a computer readable medium. As such, the claims are directed to the statutory categories of method, machine, and product of manufacture.
Step 2A Prong 1: The claimed limitations recite, inter alia: iteratively performing… a plurality of steps until a stopping criterion is satisfied, the plurality of steps comprising: Under its broadest reasonable interpretation in light of the specification, this amounts to no more than a mental process comprising determination/judgement, as performing steps until sufficiency is achieved in performable by a human being.
The claims further recite: generating, a plurality of pseudo-labels for the set of multi- sliced and multi-modal unlabeled samples in the second dataset for the second classification task; Mental process. Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process, as generating labels for unlabeled image samples is easily performable by a human with the aid of pen and paper.
The claims further recite: filtering an optimum fraction of the set of unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task based on a model confidence for the generated plurality of pseudo-labels; Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process comprising determination/judgement, as filtering based on confidence values for the aforementioned generated labels is easily performable by a human being with the aid of pen and paper.
The claims further recite: augmenting the optimum fraction of the set of unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task with corresponding pseudo-labels from the plurality of pseudo-labels; Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process comprising determination/judgement, as augmentation of the unlabeled dataset with the prior generated labels is easily performable by a human being with the aid of pen and paper.
The claims further recite: fine tuning the fuzzy deep learning architecture using (a) the set of labeled multi-sliced and multi-modal samples of the plurality of three-dimensional images in the second dataset and (b) the optimum fraction of the set of multi-sliced and multi-modal unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task augmented with corresponding pseudo-labels from the plurality of pseudo- labels; Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process, as fine tuning is a process which inherently comprises determination/judgement, and so is directed to a mental process.
The claims further recite: generating, a plurality of pseudo-labels for a remaining set of multi-sliced and multi-modal unlabeled samples in the second dataset for the second classification task; and Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process, as generating labels for unlabeled image samples is easily performable by a human with the aid of pen and paper.
The claims further recite: performing steps of training the fuzzy deep learning architecture till finetuning for the remaining set of multi-sliced and multi-modal unlabeled samples in the second dataset for the second classification task. Under its broadest reasonable interpretation in light of the specification, this amounts to no more than an abstract mental process, as performing steps until a task is done is a process which inherently comprises determination/judgement, and so is directed to a mental process.Step 2A Prong 2: The judicial exception is not sufficiently integrated into a practical application. The additional elements recited in claims 1, 8, and 15 recite: receiving, via one or more hardware processors, a first dataset and a second dataset comprising a plurality of three-dimensional images, wherein the plurality of three-dimensional images comprise a plurality of slices and a plurality of modalities, and wherein the first dataset comprises a set of labeled multi-sliced and multi-modal samples of plurality of three-dimensional images related to a first classification task and the second dataset comprises (i) a set of labeled multi-sliced and multi-modal samples and (ii) a set of unlabeled multi-sliced and multi-modal samples related to a second classification task; This amounts to no more than mere data gathering (See MPEP 2106.05(g)).
The claims further recite: inputting, via the one or more hardware processors, the first dataset to fuzzy deep learning architecture which is pretrained using the first dataset for the first classification task, wherein the fuzzy deep learning architecture comprises (i) a backbone comprising a plurality of convolutional blocks and (ii) a plurality of fuzzy layers, and (iii) a classification head comprising a plurality of average pooling layers followed by a plurality of fully connected layers and a plurality of max- pool layers, and wherein each of the plurality of fuzzy layers comprises an ordered weighted averaging (OWA) layer, a sorting function, and a weighted aggregation layer; and This amounts to no more than insignificant extra solution activity of data gathering inputs, as the claimed limitation involves selecting a particular data source or type of data to be manipulated (See 2106.05(g)).
The claims further recite: training, the fuzzy deep learning architecture using the set of multi-sliced and multi-modal labeled samples in the second dataset; This amounts to no more than a recitation of the words “Apply it” in a form of high level of generality (See MPEP 2106.05(f))Step 2B: The claims do not contain significantly more than the judicial exceptions. The additional elements of receiving, via one or more hardware processors, a first dataset and a second dataset comprising a plurality of three-dimensional images, wherein the plurality of three-dimensional images comprise a plurality of slices and a plurality of modalities, and wherein the first dataset comprises a set of labeled multi-sliced and multi-modal samples of plurality of three-dimensional images related to a first classification task and the second dataset comprises (i) a set of labeled multi-sliced and multi-modal samples and (ii) a set of unlabeled multi-sliced and multi-modal samples related to a second classification task; amounts to no more than mere data gathering (See MPEP 2106.05(g)).
inputting, via the one or more hardware processors, the first dataset to fuzzy deep learning architecture which is pretrained using the first dataset for the first classification task, wherein the fuzzy deep learning architecture comprises (i) a backbone comprising a plurality of convolutional blocks and (ii) a plurality of fuzzy layers, and (iii) a classification head comprising a plurality of average pooling layers followed by a plurality of fully connected layers and a plurality of max- pool layers, and wherein each of the plurality of fuzzy layers comprises an ordered weighted averaging (OWA) layer, a sorting function, and a weighted aggregation layer; and This amounts to no more than insignificant extra solution activity of data gathering inputs, as the claimed limitation involves selecting a particular data source or type of data to be manipulated (See 2106.05(g)). This also amounts to no more than storing or retrieving data from memory, and is known to be well-understood, routine, and conventional activity in the art (See MPEP 2106.05(d) iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc.)
The claims further recite: training, the fuzzy deep learning architecture using the set of multi-sliced and multi-modal labeled samples in the second dataset; This amounts to no more than a recitation of the words “Apply it” in a form of high level of generality (See MPEP 2106.05(f))
Considering the additional elements individually and in combination, the claims are directed to the judicial exceptions without significantly more.
Claims 2 and 9
Step 1: Claims 2 and 9 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: The claim merely narrows the previously recited abstract limitations. For the reasons described above with respect to Claim 1, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and does not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper and mathematical concepts that are achievable through mathematical computation. Step 2A Prong 2: The claim recites the additional element of: wherein each of the plurality of convolutional blocks comprises a convolutional neural network (CNN) layer followed by rectified linear activation unit (ReLu), and a first, a second and a fourth convolutional block from the plurality of convolutional blocks comprises a max-pool layer with a down- sampling factor of 2. Which amounts to no more than linking to a particular field of use or technological environment. This limitation amounts the CNN to that of a particular type, and thus is ineligible for that reason (See MPEP 2106.05(h)).Step 2B: The claims do not contain significantly more than the judicial exception.
Claims 3 and 10
Step 1: Claims 3 and 10 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: The claimed limitations recite, inter alia: performing a max-pool operation across each of the set of input features of each of the plurality of slices of the plurality of three- dimensional images to obtain a set of max-pool input features; and From available literature, a max pool process entails using a filter to select maximum values in a feature map. This is a mathematical calculation, and so entails a mental process.
The claims further recite: inputting the set of max-pool input features to a corresponding fully connected layer from a plurality of fully connected layers of the classification head followed by a sigmoid function to obtain a set of probabilities for each image from the plurality of three-dimensional images. Plugging in the max-pooled inputs to obtain a set of probabilities in images. Furthermore, using a sigmoid function to obtain a set of probabilities. This comprises a mathematical process easily performable by the human mind.Step 2A Prong 2: The judicial exception is not sufficiently integrated into a practical application. The additional elements of the claims recite: obtaining a first subset of input features of each slice of the plurality of three-dimensional images for the first classification task from each intermediate convolutional block from the plurality of convolutional blocks; Which amounts to no more than mere data gathering (See MPEP 2106.05(g))
The claims further recite: obtaining a set of input features of each slice of the plurality of three-dimensional images for the first classification task by up-sampling and adding the first subset of input features to a second subset of input features of each slice of the plurality of three-dimensional images for the first classification task obtained from a previous block; Which amounts to no more than mere data gathering (See MPEP 2106.05(g))
The claims further recite: applying a corresponding feature from the set of input features of each slice of the plurality of three-dimensional images for the first classification task to each of the plurality of fuzzy layers; which amounts to no more than reciting the words “Apply it” or a variation thereof in place of a solution (See MPEP 2106.05(f)).
Step 2B: The claims do not contain significantly more than the judicial exceptions. The additional elements of obtaining a first subset of input features of each slice of the plurality of three-dimensional images for the first classification task from each intermediate convolutional block from the plurality of convolutional blocks; amounts to no more than mere data gathering (See MPEP 2106.05(g)) obtaining a set of input features of each slice of the plurality of three-dimensional images for the first classification task by up-sampling and adding the first subset of input features to a second subset of input features of each slice of the plurality of three-dimensional images for the first classification task obtained from a previous block; Which amounts to no more than mere data gathering (See MPEP 2106.05(g)) and applying a corresponding feature from the set of input features of each slice of the plurality of three-dimensional images for the first classification task to each of the plurality of fuzzy layers; which amounts to no more than reciting the words “Apply it” or a variation thereof in place of a solution (See MPEP 2106.05(f)).
Considering the additional elements individually and in combination, the claims are directed to the judicial exceptions without significantly more.
Claims 4 and 11
Step 1: Claims 4 and 11 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: The claim merely narrows the previously recited abstract limitations. For the reasons described above with respect to Claim 1, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and does not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper and mathematical concepts that are achievable through mathematical computation.
Step 2A Prong 2: The claims recite the additional element of each of the plurality of fuzzy layers augments an incoming information to each corresponding feature in the set of input features, and wherein the incoming information is augmented by creating a set of input feature maps that are created by sorting a plurality of channels in descending order. This amounts to no more than a high-level recitation that are insignificant extra solution activity of selecting particular data source or type of data to be manipulated (See MPEP 2106.05(g)).Step 2B: The claims do not contain more than the judicial exception. The additional element of each of the plurality of fuzzy layers augments an incoming information to each corresponding feature in the set of input features, and wherein the incoming information is augmented by creating a set of input feature maps that are created by sorting a plurality of channels in descending order. This is known to be well-understood, routine, and conventional activity in the art, as it is similar to arranging a hierarchy, sorting information, eliminating less restrictive pricing information, and determining a price (MPEP 2106.05(d)(II)).
Claims 5 and 12
Step 1: Claims 5 and 12 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: The claim merely narrows the previously recited abstract limitations. For the reasons described above with respect to Claim 1, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and does not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper and mathematical concepts that are achievable through mathematical computation.Step 2A Prong 2: The claims recite the additional element of: the plurality of fuzzy layers are one dimensional and two-dimensional based on type of a weighting vector. This amounts to no more than linking a limitation to a particular field of use and technological environment (See MPEP 2106.05(h)).
Step 2B: The claims do not contain more than the judicial exception.
Claims 6 and 13
Step 1: Claims 6 and 13 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: The claim merely narrows the previously recited abstract limitations. For the reasons described above with respect to Claim 1, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and does not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper and mathematical concepts that are achievable through mathematical computation.Step 2A Prong 2: The claims recite the additional element of: the stopping criteria indicates a saturation in classification performance of the fuzzy deep learning architecture. This amounts to no more than linking a limitation to a particular field of use and technological environment (See MPEP 2106.05(h)).Step 2B: The claims do not contain more than the judicial exception.
Claims 7 and 14
Step 1: Claims 7 and 14 recites a method and a system. As such, the claims are directed to the statutory categories of method and a machine.
Step 2A Prong 1: TheStep 2A Prong 2: The claims recite the additional element of: the trained fuzzy deep learning architecture is used to learn the one or more relevant features of the plurality of three-dimensional images for the second classification task in accordance with task adaption involving knowledge transfer from the first classification task to the second classification task. This amounts to no more than linking a particular field of use or technological environment (See MPEP 2106.05(h)(vi. Limiting the abstract idea of collecting information, analyzing it, and displaying certain results of the collection and analysis).
Step 2B: The claims do not contain more than the judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over YEREBAKAN et al. (US 20250322641 A1, hereinafter Yerebakan), in view of Cella et al. (US 20250384341 A1, hereinafter Cella), in further view of Ganju et al. (US 20240045418 A1, hereinafter Ganju).
Regarding Claim 1, Yerebakan teaches the processor implemented method of Claim 1 (Paragraph [0213] The computing unit can comprise hardware and/or software. The hardware can comprise, for example, one or more processor) including:
receiving, via one or more hardware processors, a first dataset and a second dataset comprising a plurality of three-dimensional images, wherein the plurality of three-dimensional images comprise a plurality of slices and a plurality of modalities, and (Paragraph [0049] A first step is directed to receiving a target medical image series of a patient at a first point in time. A further step is directed to selecting a reference medical image study from a plurality of candidate medical image studies based on a comparison of the target body region with a plurality of candidate body regions, wherein each of the plurality of candidate body regions corresponds to one of the plurality of candidate medical image studies and each candidate medical image study comprises a plurality of candidate medical image series of the patient at a second point in time. Paragraph [0050] In particular, the medical image series can be three-dimensional images. Paragraph [0052] In particular, the type of a medical image series can be characterized by the modality used for creating the medical image and by the body region that is subject of the medical image series. Paragraph [0061] For instance, the region of instance may be an image slice of the target medical image series which may also be called “target slice”. Paragraph [0159] According to an aspect, the step of selecting at least one of the candidate medical image studies as reference medical image study comprises identifying, from the candidate medical image studies, a plurality of relevant medical image studies based on the comparison of the body regions, providing an indication of the plurality of relevant medical image studies to a user via a user interface, receiving a user selection via the user interface, the user selection indicating at least one of the relevant medical image studies, selecting the at least one indicated relevant medical image study as the reference medical image study. (Two datasets of medical images in multiple modalities (see also paragraph [0055] for further reading on receiving images in different modalities) and slices, one target which is received at a first point in time, and a reference which is received at a second point in time. The first image set (target set) is received, and the second image set (reference set) is received via user selection of a second set from a reference medical image study (Paragraph [0159])))
wherein the first dataset comprises a set of labeled multi-sliced and multi-modal samples of plurality of three-dimensional images related to a first classification task and the second dataset comprises (i) a set of labeled multi-sliced and multi-modal samples and (ii) a set of unlabeled multi-sliced and multi-modal samples related to a second classification task; (Paragraph [0174] According to an aspect, the reference medical image study comprises one or more annotations corresponding to the reference image series. The method may further comprise obtaining, from the one or more annotations, at least a reference annotation relevant for the target medical image series and/or the region of interest, and providing the reference annotation, the reference annotation preferably comprising one or more first words. Paragraph [0175] Preferably, providing the reference annotation may comprise annotating the target medical image series and/or the region of interest with the at least one reference annotation. (The reference dataset is annotated, and the target dataset is not))
(ii) generating, a plurality of pseudo-labels for the set of multi- sliced and multi-modal unlabeled samples in the second dataset for the second classification task; (Paragraph [0179] According to the above aspect, prior annotations may automatically be provided which may facilitate the follow-up reading process by the user. By automatically annotating the target medical image series and/or the region of interest, the user can directly compare prior annotations with image features of the target medical image series Paragraph [0186] With that, based on an automated annotation of the target medical image series, corresponding sections of a previous medical report may be identified which may be relevant for reading the target medical image series. (Automatic annotations are applied to the target image dataset, which are later checked))
(iii) filtering an optimum fraction of the set of unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task based on a model confidence for the generated plurality of pseudo-labels; (Paragraph [0182] Further, determining a corresponding (or target) position or determining, for each position, whether or not the position has a corresponding position in the target medical image series or the region of interest, may be based on an image similarity evaluation. Specifically, these steps may comprise obtaining, for each annotation, at least one reference image patch of the reference medical image series, the reference image patch being located relative to the position of the annotation. Further, these steps may comprise determining, for each of a plurality of target image patches in the target medical image series and/or the region of interest, a local image similarity metric indicating a degree of local image similarity between the reference image patch and the respective target image patch. Further, these steps may comprise selecting a matching image patch from the plurality of target image patches based on the similarity metrics (if possible) and determining the corresponding (or target) position based on the matching image patch. (Determining if an applicable image position exists in the target image set (filtering a fraction of unlabeled images) based on the confidence (similarity score) of the applied annotation. The filter is applied when determining the target position and whether or not if such a position even exists in the image.))
(iv) augmenting the optimum fraction of the set of unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task with corresponding pseudo-labels from the plurality of pseudo-labels; (Paragraph [0178] Annotating the target medical image series and/or the region of interest may comprise annotating the target medical image series and/or the region of interest with the at least one reference annotation at the determined corresponding location. Paragraph [0184] For instance, the target patch having the highest degree of local image similarity with the reference patch may be selected as the matching image patch. Further, a matching image patch may be identified if the degree of local image similarity is above a predefined threshold. If no target image patch can be identified the degree of local image similarity of which is above the predefined threshold, the associated annotation has no correspondence in the target medical image series and may therefore not be applicable to the target medical image series. (Labeling unlabeled samples with a labeled dataset serving as a basis. This serves to augment the unlabeled samples by applying annotations (labels) which are then later able to be scored based on similarity. These labels are tentative (and act as pseudo-labels) in the fact that they are applied first, then checked based on a reference data set to ensure they are confidently applied with a similarity score. If not, then the annotation is not applied.))
(v) fine tuning the fuzzy deep learning architecture using (a) the set of labeled multi-sliced and multi-modal samples of the plurality of three-dimensional images in the second dataset and (Paragraph [0383] According to an embodiment, a general training scheme may take the following form. Firstly, a trained function TF is received. The trained function may already be pre-trained or not having been trained at all. Next, a training data set may be provided, which comprises a plurality of images and predetermined degrees of similarity indicating a similarity between the images comprised in the training image data set. In particular, the images comprised in the training image data set may have been extracted from medical image series of one or more patients. The images comprised in the training image data set may then be inputted into the trained function TF in order to determine, by the trained function TF, degrees of similarity respectively indicative of a similarity between image pairs of the training image data set. The thus determined degrees of similarity may then be compared with the predetermined degrees of similarity. Finally, based on this comparison, the trained function TF may be adjusted and the thus adjusted trained function TF may be provided for further training or deployment. Paragraph [0387] In subsequent step T30, the first, second and third medical images are input into the trained function TF, which, at step T40 determines a first degree of similarity between the first medical image and the second medical image and a second degree of similarity between the first medical image and the third medical image. (Adjustment (fine-tuning) of an already trained model with labeled images (annotations) extracted from a medical series of images of a patient))
(b) the optimum fraction of the set of multi-sliced and multi-modal unlabeled samples of the plurality of three-dimensional images in the second dataset for the second classification task augmented with corresponding pseudo-labels from the plurality of pseudo- labels; (Paragraph [0386] The two-dimensional images are characterized in that (are chosen such that) the second medical image has a greater similarity to the first medical image than the third medical image has to the first medical image. In other words, the first medical image can be conceived as an anchor while the second medical image is the positive image, and the third medical image is the negative image. (Fine-tuning based on an optimum fraction (positive and negative image training examples) of images))
(vi) generating, a plurality of pseudo-labels for a remaining set of multi-sliced and multi-modal unlabeled samples in the second dataset for the second classification task; and (Paragraph [0179] According to the above aspect, prior annotations may automatically be provided which may facilitate the follow-up reading process by the user. By automatically annotating the target medical image series and/or the region of interest, the user can directly compare prior annotations with image features of the target medical image series [0398] The image data of the reference medical image series RMIS relative to the selected candidate location may be the basis for generating the display data in step S70. (The fine-tuning process results in data that may be used to automatically apply annotations for image data))
Yerebakan does not teach: inputting, via the one or more hardware processors, the first dataset to fuzzy deep learning architecture which is pretrained using the first dataset for the first classification task, wherein the fuzzy deep learning architecture comprises (i) a backbone comprising a plurality of convolutional blocks and (ii) a plurality of fuzzy layers, and (iii) a classification head comprising a plurality of average pooling layers followed by a plurality of fully connected layers and a plurality of max- pool layers, and wherein each of the plurality of fuzzy layers comprises an ordered weighted averaging (OWA) layer, a sorting function, and a weighted aggregation layer; and (v) fine tuning the fuzzy deep learning architecture using
In the same field of endeavor, Cella teaches: inputting, via the one or more hardware processors, the first dataset to fuzzy deep learning architecture which is pretrained using the first dataset for the first classification task, wherein the fuzzy deep learning architecture comprises (Paragraph [0387] Embodiments of the present disclosure, including ones involving expert systems, self-organization, machine learning, artificial intelligence, and the like, may benefit from the use of a neural net, such as a neural net trained for pattern recognition, for classification of one or more parameters, characteristics, or phenomena, for support of autonomous control, and other purposes. References to a neural net throughout this disclosure should be understood to encompass a wide range of different types of neural networks, machine learning systems, artificial intelligence systems, and the like, such as convolutional neural networks, hybrids of neural networks with other expert systems (e.g., hybrid fuzzy logic-neural network systems) Paragraph [0546] Components of domain-specific and general artificial intelligence models 3718 may include artificial intelligence building blocks, such as a component that detects and translates between languages, or a component that delivers highly personalized customer recommendations. (The model of Cella contains the limitations of the instant application. A pretrained architecture [2436], in embodiments, a hybrid fuzzy/convolutional deep learning architecture [0387], a classification layer [2436] and layers which aggregate information [1051]. The model is pretrained in order to classify data.))
(i) a backbone comprising a plurality of convolutional blocks and (Paragraph [0546] Components of domain-specific and general artificial intelligence models 3718 may include artificial intelligence building blocks, such as a component that detects and translates between languages, or a component that delivers highly personalized customer recommendations. Paragraph [0547] For example, attributes of a task involving 2D object classification may be searched in the artificial intelligence store 3770 and its metadata 3724 to reveal that an artificial intelligence model 3718 suitable for a task involving 2D object classification may be a convolutional neural network. (Convolutional network comprising blocks))
(ii) a plurality of fuzzy layers, and (Paragraph [0425] In embodiments, methods and systems described herein that involve an expert system or self-organization capability may use a neuro-fuzzy network, such as involving a fuzzy inference system in the body of an artificial neural network. Depending on the type, several layers may simulate the processes involved in a fuzzy inference, such as fuzzification, inference, aggregation and defuzzification. Embedding a fuzzy system in a general structure of a neural net as the benefit of using available training methods to find the parameters of a fuzzy system. (The model is comprised of fuzzy layers))
(iii) a classification head comprising a plurality of average pooling layers followed by a plurality of fully connected layers and a plurality of max- pool layers, and wherein each of the plurality of fuzzy layers comprises an ordered weighted averaging (OWA) layer, a sorting function, and a weighted aggregation layer; and (Paragraph [2411] Different pooling functions may be used in the pooling layer, including max pooling, average pooling, and l2-norm pooling. Paragraph [2436] In various embodiments, the RPA module 17916 uses one or more artificial intelligence modules 17904 that are specifically designed and/or trained for the workflow. In various embodiments, the RPA module 17916 uses one or more pretrained artificial intelligence modules 17904 to perform the processing of the workflow. The adaptation can involve applying transfer learning to an artificial intelligence module 17904 (e.g., more specifically training one or more classification layers in a classification portion of the NLP machine learning model while holding other portions of the NLP machine learning model constant). (The model comprising pooling layers such as max-pool and classification layers)) and
(v) fine tuning the fuzzy deep learning architecture using (Paragraph [2756] In some examples, the learning models may train first from a larger corpus of training data (e.g., public training data set) and then undergo a fine-tuning process that trains with a specialized data set that is particular to digital enterprise assets. (Fine-tuning the deep learning model))
It would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the concept of using a hybrid fuzzy-convolutional neural network to process a dataset, where the model was pre-trained on a separate dataset, and with the model comprising convolutional blocks, fuzzy layers, and a classification head followed by pooling layers as taught by Cella into Yerebakan as both are in the same field of machine learning and image recognition, and this combination would allow for end users to access higher amounts of resources than they otherwise would be able to (Cella Paragraph [0003]).
The combination of Yerebakan and Cella does not teach: iteratively performing, via the one or more hardware processors, a plurality of steps until a stopping criterion is satisfied, the plurality of steps comprising: (i) training, the fuzzy deep learning architecture using the set of multi-sliced and multi-modal labeled samples in the second dataset; (vii) performing steps of training the fuzzy deep learning architecture till finetuning for the remaining set of multi-sliced and multi-modal unlabeled samples in the second dataset for the second classification task.
In the same field of endeavor, Ganju teaches iteratively performing, via the one or more hardware processors, a plurality of steps until a stopping criterion is satisfied, the plurality of steps comprising: (Paragraph [0459] In at least one embodiment, customer dataset 3506 may be applied to initial model 3504 any number of times, and ground truth data may be used to update parameters of initial model 3504 until an acceptable level of accuracy is attained for refined model 3512. (Performing steps until a stopping condition (accuracy) is reached))
(i) training, the fuzzy deep learning architecture using the set of multi-sliced and multi-modal labeled samples in the second dataset; (Paragraph [0395] In at least one embodiment, in some examples, labeled clinic data 3112 (e.g., annotations provided by a clinician, doctor, scientist, technician, etc.) may be used as ground truth data for training a machine learning model. Paragraph [0416] In at least one embodiment, deployment pipelines 3210 may include any number of applications that may be sequentially, non-sequentially, or otherwise applied to imaging data (and/or other data types) generated by imaging devices, sequencing devices, genomics devices, etc. —including AI-assisted annotation, as described above. Paragraph [0439] In at least one embodiment, and with reference to FIGS. 34A-34B, deployment system 3106 may be implemented as one or more virtual instruments to perform different functionalities—such as image processing, segmentation, enhancement, AI, visualization, and inferencing—with imaging devices (e.g., CT scanners, X-ray machines, MRI machines, etc.), sequencing devices, genomics devices, and/or other device types. In at least one embodiment, system 3200 may allow for creation and provision of virtual instruments that may include a software-defined deployment pipeline 3210 that may receive raw/unprocessed input data generated by a device(s) and output processed/reconstructed data. Paragraph [0462] In at least one embodiment, imaging applications may include software tools that help user 3510 to identify, as a non-limiting example, a few extreme points on a particular organ of interest in raw images 3534 (e.g., in a 3D MRI or CT scan) and receive auto-annotated results for all 2D slices of a particular organ. In at least one embodiment, results may be stored in a data store as training data 3538 and used as (for example and without limitation) ground truth data for training. (Training data may include imaging data taken with different imaging devices, such as CT scans, X-Ray, etc. These are different imaging modalities. Further, in some embodiments these images taken and used as training data are pre-annotated (labeled). In addition, annotated results in the form of two-dimensional slices may be used as further training data))
(vii) performing steps of training the fuzzy deep learning architecture till finetuning for the remaining set of multi-sliced and multi-modal unlabeled samples in the second dataset for the second classification task. (Paragraph [0459] In at least one embodiment, customer dataset 3506 may be applied to initial model 3504 any number of times, and ground truth data may be used to update parameters of initial model 3504 until an acceptable level of accuracy is attained for refined model 3512. (The steps can be reiterated any number of times until a more acceptable level of accuracy is attained))
It would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated performing steps until a stopping criterion is reached, training a neural network with multi-sliced and multi-modal images and performing steps to ultimately fine tune the model as taught by Ganju into the combination of Yerebakan and Cella as all three are in the fields of machine learning and data recognition and the combination would address the need to detect abnormalities when monitoring sensitive areas (Ganju Paragraphs [0002] - [0003]).
Regarding Claims 8 and 15, claims 8 and 15 are system and computer readable storage claims corresponding to the method of Claim 1. As such, these claims are rejected for the same reasons.
Regarding Claim 2, the combination of Yerebakan, Cella, and Ganju teaches all the limitations as applied in Claim 1, including: wherein each of the plurality of convolutional blocks comprises a convolutional neural network (CNN) layer followed by rectified linear activation unit (ReLu), and a first, a second and a fourth convolutional block from the plurality of convolutional blocks comprises a max-pool layer with a down- sampling factor of 2. (Cella Paragraph [2396] A third example activation function is the rectified linear unit (ReLU) function. The ReLU function takes a real-valued input and thresholds it above zero (i.e., replacing negative values with zero): Paragraph [2411] Different pooling functions may be used in the pooling layer, including max pooling, average pooling, and l2-norm pooling. (An activation function particularly comprising a RELU function. Further, one possible embodiment of pooling in the reference is discloses as max pooling.))
Regarding Claim 9, claim 9 is a system claim corresponding to the method of Claim 2, and is rejected for the same reasons.
Regarding Claim 3, the combination of Yerebakan, Cella, and Ganju teaches the invention as claimed in Claim 1, including: (i) obtaining a first subset of input features of each slice of the plurality of three-dimensional images for the first classification task from each intermediate convolutional block from the plurality of convolutional blocks; (Ganju Paragraph [0423] In at least one embodiment, AI services 3218 may leverage AI system 3224 to execute machine learning model(s) (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and/or other inferencing tasks. In at least one embodiment, applications of deployment pipeline(s) 3210 may use one or more of output models 3116 from training system 3104 and/or other models of applications to perform inferencing on imaging data (e.g., DICOM data, RIS data, CIS data, REST compliant data, RPC data, raw data, etc.) Paragraph [0435] In at least one embodiment, deployment pipeline 3210A of FIG. 33 may include CT scanner 3302 generating imaging data of a patient or subject. In at least one embodiment, imaging data from CT scanner 3302 may be stored on a PACS server(s) 3304 associated with a facility housing CT scanner 3302. In at least one embodiment, PACS server(s) 3304 may include software and/or hardware components that may directly interface with imaging modalities (e.g., CT scanner 3302) at a facility. Paragraph [0462] In at least one embodiment, imaging applications may include software tools that help user 3510 to identify, as a non-limiting example, a few extreme points on a particular organ of interest in raw images 3534 (e.g., in a 3D MRI or CT scan) and receive auto-annotated results for all 2D slices of a particular organ. (The method of Ganju extracts features from images receieved in multiple modalities, such as from a CT Scan. Further, the method is also able to be used to detect points of interest in two-dimensional slices of an organ.))
(ii) obtaining a set of input features of each slice of the plurality of three-dimensional images for the first classification task by up-sampling and adding the first subset of input features to a second subset of input features of each slice of the plurality of three-dimensional images for the first classification task obtained from a previous block; (Cella Paragraph [2686] In some cases, the training data set may include both authentic training data and synthetic training data that is based on the authentic training data (e.g., both a real-world image and a modified version of the real-world image that has been adjusted in brightness, contrast, size, resolution, scale, shape, aspect ratio, color depth, or the like). (The input features includes upscaled images derived from real world examples.))
(iii) applying a corresponding feature from the set of input features of each slice of the plurality of three-dimensional images for the first classification task to each of the plurality of fuzzy layers; (Cella Paragraph [0410] A convolutional neural network may be used for recognition within images and video streams, such as for recognizing a type of machine in a large environment using a camera system disposed on a mobile data collector, such as on a drone or mobile robot. In embodiments, a convolutional neural network may be used to provide a recommendation based on data inputs, including sensor inputs and other contextual information, such as recommending a route for a mobile data collector. Paragraph [0591] In embodiments, the cognitive processes system 3972 may analyze input data (e.g., blueprints, image scans, 3D data) to classify rooms, pathways, equipment, and the like to assist in the generation of the 3D representation. (Using the model layers to identify features in images))
(iv) performing a max-pool operation across each of the set of input features of each of the plurality of slices of the plurality of three- dimensional images to obtain a set of max-pool input features; and (Yerebakan Paragraph [0132] Based on a fully convolutional Siamese network at least one convolution processing and at least one pooling processing may be executed on, e.g., a first slice of the first image data set, thus obtaining image features (or the image descriptor) of that first image slice. Paragraph [0375] Alternative network arrangements may be used, for example, a 3D Very Deep Convolutional Network (3D-VGGNet), wherein a VGGNet stacks many layer blocks containing narrow convolutional layers followed by max pooling layers. (Performing a pooling operation in order to extract input features of the images))
(v) inputting the set of max-pool input features to a corresponding fully connected layer from a plurality of fully connected layers of the classification head followed by a sigmoid function to obtain a set of probabilities for each image from the plurality of three-dimensional images. (Paragraph [0484] In some examples, the trained neural network may be part of a Siamese neural network trained to output a similarity metric indicating a degree of patch similarity between two input portions of medical imaging data. A similarity measure (e.g., Euclidean distance, cosine similarity, or the like) may be determined between the first and second patch descriptors, and this may be passed through an activation function (e.g., sigmoid function or the like) to produce a similarity metric indicating the degree of patch similarity between the target image patch RM-202 and the reference image patch RM-302. (Inputting the features in order to obtain a metric (in this case, similarity) using a sigmoid function))
Regarding Claim 10, claim 10 is a system claim which corresponds to the method of Claim 3, and so is rejected for the same reasons.
Regarding Claim 4, the combination of Yerebakan, Cella, and Ganju teaches all the limitations as applied in Claim 3, including:
each of the plurality of fuzzy layers augments an incoming information to each corresponding feature in the set of input features, and wherein the incoming information is augmented by creating a set of input feature maps that are created by sorting a plurality of channels in descending order. (Cella Paragraph [0307] The terms classify, classifying, classification, categorization, categorizing, categorize (and similar terms) as utilized herein should be understood broadly. Without limitation to any other aspect or description of the present disclosure, classifying a condition or item may include actions to sort the condition or item into a group or category based on some aspect, attribute, or characteristic of the condition or item where the condition or item is common or similar for all the items placed in that classification, Paragraph [0387] Embodiments of the present disclosure, including ones involving expert systems, self-organization, machine learning, artificial intelligence, and the like, may benefit from the use of a neural net, such as a neural net trained for pattern recognition, for classification of one or more parameters, characteristics, or phenomena, for support of autonomous control, and other purposes. (The inputted information is classified, which the reference broadens to include sorting))
Regarding Claim 11, claim 11 is a system claim which corresponds to the method of Claim 4, and is rejected for the same reasons.
Regarding Claim 5, the combination of Yerebakan, Cella, and Ganju teaches all the limitations as applied in Claim 1, including:
the plurality of fuzzy layers are one dimensional and two-dimensional based on type of a weighting vector. (Cella Paragraph [0397] For classification problems, one output is produced (with a separate set of weights and summation units) for each target category. The value output for a category is the probability that the case being evaluated has that category. In training of an RBF, various parameters may be determined, such as the number of neurons in a hidden layer, the coordinates of the center of each hidden-layer function, the spread of each function in each dimension, and the weights applied to outputs as they pass to the summation layer. (Layer dimensions are dependent on vectors with weights.))
Regarding Claim 12, claim 12 is a system claim which corresponds to the method of Claim 5, and is rejected for the same reasons.
Regarding Claim 6, the combination of Yerebakan, Cella, and Ganju teaches all the limitations as applied in Claim 1, including:
the stopping criteria indicates a saturation in classification performance of the fuzzy deep learning architecture. (Ganju Paragraph [0459] In at least one embodiment, customer dataset 3506 may be applied to initial model 3504 any number of times, and ground truth data may be used to update parameters of initial model 3504 until an acceptable level of accuracy is attained for refined model 3512. (The stopping criteria indicates saturation of performance (accuracy)))
Regarding Claim 13, claim 13 is a system claim which corresponds to the method of Claim 6, and is rejected for the same reasons.
Regarding Claim 7, the combination of Yerebakan, Cella, and Ganju teaches all the limitations as applied in Claim 1, including: the trained fuzzy deep learning architecture is used to learn the one or more relevant features of the plurality of three-dimensional images for the second classification task in accordance with task adaption involving knowledge transfer from the first classification task to the second classification task. (Yerebakan Paragraph [0101] In particular, the trained function may comprise a neural network. Thereby, a first group of neural network layers may be applied to extract first and second feature vectors. The feature vectors may be fed as input values to a second group of network layers which serve to determine a degree of comparability based on the extracted features vectors. However, both functions of the described neural network may likewise be carried out by separated, individual neural networks. Paragraph [0126] According to an aspect, the step of identifying the at least one reference slice comprises applying a trained function on the target and reference medical image data series, wherein the trained function is adapted to determine degrees of similarities between two-dimensional medical images. Optionally, the trained function further applies a learned metric to determine degrees of similarity between two-dimensional medical images, the trained function preferably comprising a deep metric learned network. Paragraph [0263] Determining the body region 408 represented by the medical imaging data ID by inputting the one or more text strings AV, IF-314 associated with the target medical image study TS into a machine learning model 406 (e.g., a neural network) trained to determine a body region based on input such text strings, may provide for efficient and/or flexible determination of the body region 408 represented by the medical imaging data comprised in target and/or candidate image studies TS, CS. (The trained model can be leveraged to extract other features from the image or identify what body part the image was taken from))
Regarding Claim 14, claim 14 is a system claim which corresponds to the method of Claim 7, and is rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicants’ disclosure.
Zhao et al. (US 20220012859 A1) discusses processing images using neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN A CARDOSO whose telephone number is (571)272-8512. The examiner can normally be reached M-F 7:30 - 5:00, alternate Friday's off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUSTIN CARDOSO/
Patent Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights