DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-15 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 14 is drawn to functional descriptive material recorded on a “A computer program product”. Normally, the claim would be statutory. However, the broadest reasonable interpretation of a claim drawn to a “computer program product” typically covers forms of non-transitory tangible media as well as transitory propagating signals per se, making the recited claim language directed towards non-statutory subject matter such as a “signal”.
“A transitory, propagating signal … is not a “process, machine, manufacture, or composition of matter.” Those four categories define the explicit scope and reach of subject matter patentable under 35 U.S.C. § 101; thus, such a signal cannot be patentable subject matter.” (In re Nuijten, 84 USPQ2d 1495 (Fed. Cir. 2007)).
Because the full scope of the claim as properly read in light of the disclosure appears to encompass non-statutory subject matter (i.e., because the specification is silent to the exact embodiment of a computer readable medium, it is interpreted as including the ordinary and customary meaning of computer readable medium covering both non-transitory media and transitory propagating signals, etc.) the claim as a whole is non-statutory. In view of the USPTO's Interim Examination Instructions for Evaluating Subject Matter Eligibility under 35 U.S.C. 101 (the "Guidelines"), and the Official Gazette Notice (1351 OG 212, made available February 23, 2010), the examiner suggests amending the claim to include the limitation "non-transitory" in order to exclude any non-statutory subject matter. Any amendment to the claim should be commensurate with its corresponding disclosure.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8, and 14-15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kobald et al. (Jonathan Kobold. Deep Learning for lesion and thrombus segmentation from cerebral MRI. Image Processing [eess.IV]. Université Paris Saclay (COmUE), 2019. English. ⟨NNT : 2019SACLE044⟩. ⟨tel-03592570⟩ Available online: https://theses.hal.science/tel-03592570v1, hereby referred to as “Kobald”). Kobald was cited by applicant in IDS submitted on .January 4, 2024.
Consider Claim 1.
Kobald teaches:
1. (Currently Amended) An automatic method(Kobald: pages 2-3, section Introduction; And this is the starting point of this thesis. The goal is to provide an automatic method for segmenting the lesion and the thrombus. This will improve the interpretation of stroke MRI by giving numerical values for the relevant parameters instead of the rough guesses which are currently used. The methods of choice for this kind of segmentation problem are machine learning methods. This makes this thesis an intriguing subject combining knowledge from the fields of physics, machine learning and medicine. This multidisciplinary approach is also reflected in the structure of this thesis by explaining the basics from each discipline separately. Of course it is indispensable to understand what stroke is and how it works. This is explained in the medical part (section 1.1). To understand the MR images it is necessary to understand how they are generated, i.e. the physics of MRI (section 1.2.1). The medical and physical knowledge are merged in the interpretation of the MR images (section 1.2). Then machine learning and its important field deep learning are introduced in chapter 2. Together with a review of existing automatic methods for stroke MRI (section 1.3) this is the basis for exploring the stroke segmentation problem. One of the big opportunities of this thesis is that it not only allows to apply machine learning to a real problem but also the problem is so complex that it allows new insights into machine learning theory. New approaches on neural network design are elaborated in chapter 3 and their application on the stroke segmentation task is detailed in chapter 4. Altogether the subject of this thesis is very complex and encompasses three disciplines, physics, medicine and machine learning. The latter has not seen any clinical application in a stroke scenario yet. The complexity of the problems entails the further exploration of machine learning theory. And the results can be directly applied to a relevant problem of today’s medicine and will eventually find its way to clinical practice. Most thesis subjects are either of theoretical or applied nature and it is the special beauty of the stroke MRI segmentation problem that it allows to have both.)
1. the three-dimensional brain image being acquired according to at least one modality, each modality being associated with a set of images comprising a plurality of images each corresponding to a section of the three-dimensional image acquired according to the modality, along a sectional plane perpendicular to a given axis, the methodcomprising: (Kobald: page 3, section 1. Introduction, Of course it is indispensable to understand what stroke is and how it works. This is explained in the medical part (section 1.1). To understand the MR images it is necessary to understand how they are generated, i.e. the physics of MRI (section 1.2.1). The medical and physical knowledge are merged in the interpretation of the MR images (section 1.2). page 17, section 1.2. Diagnosis on MRI, Globally speaking, the spin density operator represents the macroscopic magnetization of the sample arising from the ensemble of the individual spins. The part in direction of B0 is given by the populations and the part perpendicular is given by ρ−. Page 21, section Slice Selection;)
1. supervised training of at least one primary recurrent artificial neural network configured to provide lesion prediction from an image, (Kobald: page 3, section 1. Introduction, Then machine learning and its important field deep learning are introduced in chapter 2. Together with a review of existing automatic methods for stroke MRI (section 1.3) this is the basis for exploring the stroke segmentation problem. One of the big opportunities of this thesis is that it not only allows to apply machine learning to a real problem but also the problem is so complex that it allows new insights into machine learning theory. New approaches on neural network design are elaborated in chapter 3 and their application on the stroke segmentation task is detailed in chapter 4. Altogether the subject of this thesis is very complex and encompasses three disciplines, physics, medicine and machine learning. The latter has not seen any clinical application in a stroke scenario yet. The complexity of the problems entails the further exploration of machine learning theory. And the results can be directly applied to a relevant problem of today’s medicine and will eventually find its way to clinical practice. Most thesis subjects are either of theoretical or applied nature and it is the special beauty of the stroke MRI segmentation problem that it allows to have both.)
1. each primary recurrent artificial neural network being associated with a set of training parameters and trained on a primary database including a plurality of brain images each associated with a set of information relating to the segmentation of each lesion in the image; (Kobald: pages 48-49 Chapter 2 Machine learning This chapter aims to introduce machine learning, deep learning and some of the models commonly used for bio-medical data-sets. It starts with some necessary definitions and will then introduce the basics of machine learning. Following this the concept of deep learning will be introduced and building upon that convolutional neural networks and recurrent networks. Finally the Long Short Term Memory (LSTM) [89] based models as well as U-Net variants are presented.
PNG
media_image1.png
378
668
media_image1.png
Greyscale
page 60 section 2.4.2. Neural Networks, section Recurrent Networks, In contrast to the feed forward network a recurrent neural network (RNN) can be a cyclic graph. A step can be dependent on the result of a future step, or even its own result. Also network models with a memory (i.e. ~y depends not only on ~x but also on all other inputs the model has seen in the past or future) fall under the definition of a recurrent network. This comes from the fact that ”memory“ can be represented by a self loop in the model. Both types are constructed as a sequence of operations, also called layers, with simple to calculate gradients. This structure makes training with gradient descent (section 2.2.2) quite easy. Via the chain rule for derivatives the derivative of the loss can be calculated analytically for each layer. This results in the gradient of each layer being dependent on the derivatives of the following layers. This can be seen as the error being propagated ”backwards“ through the network in order to calculate the gradients of the individual layers. Hence the name ”backpropagation“ for training of neural networks with gradient descent. There are many choices for layers. The most popular ones are dense layers (equation 2.31), convolution layers (equation 2.13) and maximum pooling layers (equation 2.14).)
1. supervised training of at least one secondary recurrent artificial neural network configured to provide a thrombus prediction(Kobald: Page 98 Chapter 4, Stroke MRI Segmentation The main goal of this work is to find the signs of stroke relevant for diagnosis on MRI using machine learning techniques. The most relevant are the lesion and the thrombus. So this translates to the task of an automatic segmentation of lesion and thrombus from stroke MRI. This chapter describes the data-set and pre-processing steps and the results achieved with different automatic methods. Figure 4.1 The size distribution of the lesions a) and thrombi b) in the CHSF stroke data-set. Most of the lesions are small which confirms that the MRI are from the hyper-acute phase of stroke. For the purpose of this figure the lesion or thrombus was calculated as the union of the multiple segmentations available for each patient. page 110-111, section 4.3 Automatic Thrombus Segmentation, subsection 4.3.1 Multi-Directional U-Net The U-Net (section 2.5.1) is a very successful method for bio-medical image segmentation. Thus it was also tested for thrombus segmentation. The models are trained from scratch on the CHSF stroke data-set. The first trained U-Nets didn’t deliver even remotely useful results. It turned out that the class imbalance between thrombus and other tissues was too massive. This lead to the sampling scheme described in section 4.1.1 which mostly fixed the class imbalance. With this sampling the U-Net started to learn that dark thin objects have to be segmented, resulting in a model which segmented all dark areas like veins. And there are so many dark areas on susceptibility weighted angiography (SWAN) (see figure 4.7 a)) that this is no useful segmentation either.)
1. each secondary recurrent artificial neural network being associated with a set of learning parameters and trained on a secondary database including a plurality of brain images each associated with a set of information relating to the segmentation of each thrombus in the image; (Kobald: page 82 section 3.1.4 BraTS For the second experiment a real data set which is renowned for its difficulty is chosen. The brain tumor segmentation (BraTS) [101] challenge is a recurring challenge attached to the MICCAI Conference. Each year the segmentation results become better, but the problem is an ongoing research. For this experiment the high grade glioma part of the BraTS 2017 data-set2 is used. It contains multi-modal MRI of 210 patients which were manually segmented by experts, i.e. a ground truth is available. On these image three different classes have to be segmented from the background. The enhancing tumor, the necrotic and non-enhancing tumor and as a third class the peritumoral oedema. This makes it an ideal real data-set for supervised learning of a multi-class segmentation task. It was also chosen to provide a publicly available reference. Pages 100-101)
1. using each primary recurrent artificial neural network trained, on each image of a primary set of images dependent on at least one set of images associated with a modality, (Kobald: pages 48-49 Chapter 2 Machine learning
PNG
media_image1.png
378
668
media_image1.png
Greyscale
page 60 section 2.4.2. Neural Networks, section Recurrent Networks, In contrast to the feed forward network a recurrent neural network (RNN) can be a cyclic graph. A step can be dependent on the result of a future step, or even its own result. Also network models with a memory (i.e. ~y depends not only on ~x but also on all other inputs the model has seen in the past or future) fall under the definition of a recurrent network. This comes from the fact that ”memory“ can be represented by a self loop in the model. Both types are constructed as a sequence of operations, also called layers, with simple to calculate gradients. This structure makes training with gradient descent (section 2.2.2) quite easy. Via the chain rule for derivatives the derivative of the loss can be calculated analytically for each layer. This results in the gradient of each layer being dependent on the derivatives of the following layers. This can be seen as the error being propagated ”backwards“ through the network in order to calculate the gradients of the individual layers. Hence the name ”backpropagation“ for training of neural networks with gradient descent. There are many choices for layers. The most popular ones are dense layers (equation 2.31), convolution layers (equation 2.13) and maximum pooling layers (equation 2.14).)
1. and merging the lesion predictions obtained to obtain a set of lesion segmentations: (Kabold: page 113-114, section 4.3 Automatic Thrombus Segmentation, Consultation with the neurologists of the CHSF revealed that the context available from a single slice of the SWAN is not enough to identify a thrombus. For example a vein with flow perpendicular to the slice produces a black spot indistinguishable from a thrombus (compare figure 4.5). This excuses the U-Net for its bad performance. To improve this segmentation results the three dimensional context needs to be included. For this three separate UNets are trained (with the retraining method above) on different orientations of the MRI. One is trained on axial slices, one on coronal slices and one on sagittal slices (see figure 4.6). The three models provide three different scores s1, s2, s3 for each voxel (figure 4.7 d)-f)) which need to be merged for a final decision. A simple majority vote would underestimate the size of the thrombus. Thus the label merging is split into two parts. At first possible thrombi are extracted and in the second part these candidates are classified into thrombus and non thrombus. For the first part a new volume V is created where each voxel is assigned the maximum max(s1, s2, s3) out of the three scores. V is then thresholded at a value of 0.4 and the resulting binary map is divided into connected components. To decide whether one component should be a thrombus or not, the scores (s1, s2, s3) are used again. If all three scores are bigger than 0.7 for one voxel in the component, i.e. s1 > 0.7 ∧ s2 > 0.7 ∧ s3 > 0.7, then the component is validated as a thrombus (figure 4.7 c)). This architecture merging multiple U-Nets along multiple directions,)
1.if the set of lesion segmentations includes at least one segmentation, selecting the segmentation with the maximum volume as the lesion segmentation; (Kobald: pages 48-49, Figure 2.1, page 60 section 2.4.2. Neural Networks, section Recurrent Networks, Page 98 Chapter 4, Stroke MRI Segmentation The main goal of this work is to find the signs of stroke relevant for diagnosis on MRI using machine learning techniques. The most relevant are the lesion and the thrombus. So this translates to the task of an automatic segmentation of lesion and thrombus from stroke MRI. This chapter describes the data-set and pre-processing steps and the results achieved with different automatic methods. Page 98, Figure 4.1 The size distribution of the lesions a) and thrombi b) in the CHSF stroke data-set. Most of the lesions are small which confirms that the MRI are from the hyper-acute phase of stroke. For the purpose of this figure the lesion or thrombus was calculated as the union of the multiple segmentations available for each patient. page 110-111, section 4.3 Automatic Thrombus Segmentation, subsection 4.3.1 Multi-Directional U-Net The U-Net (section 2.5.1))
1. using each secondary recurrent artificial neural network trained, on each image of a secondary set of images dependent on at least one set of images associated with a modality, (Kobald: pages 49-50, section 2.1.2 Working with Volumes, Volumes in the sense of this work are hypercubes of an N-dimensional space with a cartesian grid. Each position on the grid is attributed a value. Thus the volume is a finite set of values (voxels) v(i1, i2, . . . , iN ) : N N → R which are indexed through their position on the grid, i.e. by N indices i1, i2, . . . , iN with ik ∈ [1 . . . nk] . The nk are called the axis sizes of the volume. pages 87-88, section 3.2.1 Double Pass, Figure 3.10
PNG
media_image2.png
424
670
media_image2.png
Greyscale
pages 115-119, section 4.3.2. Mask R-CNN, section 4.3.3 Logic LSTM )
1.and merging the thrombus predictions obtained to obtain a set of thrombus segmentations: If the set of thrombus segmentations includes at least one segmentation, selecting the segmentation satisfying a proximity condition as the thrombus segmentation, the proximity condition depending on the lesion segmentation. (Kobald: page 116 The 3D U-Net architecture is almost identical to the U-Net used above, just that 3D convolutions are used on 3D inputs instead of 2D. It turns out that the 3D U-Net shares the problems of the U-Net. It only manages to detect black spots but it doesn’t manage to differentiate between thrombus and non thrombus. On average the 3D U-Net detects 208 objects on the test set which is worse than the U-net. Certainly this could be improved by using the same cross validation and retraining scheme as for the U-Net. But one 3D U-Net model already needs around 4 days to train on an NVIDIA Tesla V100 GPU so this is not feasible. Page 117 It turns out that adding the lesion indeed improves the results significantly. The thrombus needs to be in the blood stream which supplies the lesion region. For all patients in the data base this is in front and below of the lesion, as well as between the outer border of the lesion and the mid-sagittal plane. To avoid calculating the position of the mid-sagittal plane it is assumed that the thrombus is not further inward than 50 voxels from the inner edge of the lesion. This excludes the unaffected hemisphere and is far enough for all thrombi. Sometimes the thrombus can be inside the lesion, so the selection rule with respect to the lesion is: Thrombus selection by lesion position The valid location for the thrombus is inside a cube given by the lesion location. It is below the upper edge of the lesion, in front of the rear edge of the lesion and between the outer edge of the lesion and 50 voxels inward of the inner edge of the lesion. The inner edge is the one closest to the mid sagittal plane (which can be replaced by the centre of the MRI volume for calculation purposes). This selection allows to discard many false positives. Using the lesion the number of false positives drops massively for all models (table 4.3 lower part). But the relative performance of the models stays the same. The UNet and 3D U-Net still detect more than 40 false positives which is far from a reliable detection. The merged logic LSTM model Th 1 & Th 2 improves to only 1.5 false positive objects with a detection rate of 93%.)
Consider Claim 2.
Kobald teaches: 2. (Currently Amended) The method according to claim 1, wherein the three- dimensional image is acquired by MRI. (Kobald: Page 98 Chapter 4, Stroke MRI Segmentation The main goal of this work is to find the signs of stroke relevant for diagnosis on MRI using machine learning techniques. The most relevant are the lesion and the thrombus. So this translates to the task of an automatic segmentation of lesion and thrombus from stroke MRI. This chapter describes the data-set and pre-processing steps and the results achieved with different automatic methods. Figure 4.1 The size distribution of the lesions a) and thrombi b) in the CHSF stroke data-set. Most of the lesions are small which confirms that the MRI are from the hyper-acute phase of stroke. For the purpose of this figure the lesion or thrombus was calculated as the union of the multiple segmentations available for each patient. page 110-111, section 4.3 Automatic Thrombus Segmentation, subsection 4.3.1 Multi-Directional U-Net The U-Net (section 2.5.1) is a very successful method for bio-medical image segmentation. Thus it was also tested for thrombus segmentation. The models are trained from scratch on the CHSF stroke data-set. The first trained U-Nets didn’t deliver even remotely useful results. It turned out that the class imbalance between thrombus and other tissues was too massive. This lead to the sampling scheme described in section 4.1.1 which mostly fixed the class imbalance. With this sampling the U-Net started to learn that dark thin objects have to be segmented, resulting in a model which segmented all dark areas like veins. And there are so many dark areas on susceptibility weighted angiography (SWAN) (see figure 4.7 a)) that this is no useful segmentation either.)
Consider Claim 3.
Kobald teaches: 3. (Currently Amended) The method according to claim 2, wherein the three- dimensional image is acquired by MRI according to a first susceptibility weighted angiography SWAN modality, a second phase modality of the radiofrequency signal of the susceptibility weighted angiography SWAN, a third time-of-flight ToF modality, a fourth diffusion weighted images DWI modality and a fifth diffusion weighted images DWI modality with exclusive application of the main magnetic field Bo. (Kobald: page 110-111, section 4.3 Automatic Thrombus Segmentation, subsection 4.3.1 Multi-Directional U-Net The U-Net (section 2.5.1) is a very successful method for bio-medical image segmentation. Thus it was also tested for thrombus segmentation. The models are trained from scratch on the CHSF stroke data-set. The first trained U-Nets didn’t deliver even remotely useful results. It turned out that the class imbalance between thrombus and other tissues was too massive. This lead to the sampling scheme described in section 4.1.1 which mostly fixed the class imbalance. With this sampling the U-Net started to learn that dark thin objects have to be segmented, resulting in a model which segmented all dark areas like veins. And there are so many dark areas on susceptibility weighted angiography (SWAN) (see figure 4.7 a)) that this is no useful segmentation either.
PNG
media_image3.png
624
660
media_image3.png
Greyscale
)
Consider Claim 4.
Kobald teaches:
4. (Currently Amended) The method according to claim 2, including a pre-processing stepcalculating a histogram on the grey levels of the voxels of the three-dimensional image acquired according to the modality; calculating a polynomial approximation of the logarithm of the histogram; (Kobald: page 102, section 4.2.1, Co-Registration and Data Format, All the MRI modalities have different resolutions. As the patient may move his head between scans the head position and inclination may be different as well. To bring all images to the same resolution the co-registration algorithm of the SPM 12 toolbox1 for Matlab is used. The co-registration target is the SWAN, as it has the highest resolution. A down-sampling of the SWAN may loose the thrombus, compared to this a quality loss from up-sampling and rotating the other images is negligible. Section 4.2.2 Normalisation, MRI is a qualitative and no quantitative measurement [103], i.e. the intensity values and range of repeated measurements of the same patient are different even though it appears to be the same image to human eyes. Across patients this variability is even higher (compare figures 4.2 a) and c)). Therefore a normalisation of the images is necessary to obtain repeatable results with automatic processing of the images. Luckily the MRIs all have a common characteristic. Most of the brain is composed of healthy tissue which is visible as a large peak in the histogram of the voxel intensities, as indicated by the red lines in figure 4.2. This peak is only overshadowed by the peak corresponding to the empty space around. Figure 4.2 Two examples of the intensity normalisation of the SWAN histogram. Note the location of the peak corresponding to the healthy brain tissue (red line) in the original histograms a) and c). Both histograms show this peak, but its location is different. This illustrates why intensity normalisation of MRI is important. Then the intensity values are divided by the peak intensity value and the normalised histograms b) and d) are obtained. Note that this histogram shows intensity values of the brain only, the very large peak around zero of the empty space around the head has been removed for readability. The histograms of other MRI modalities are very similar and the normalisation process is identical.)
4. applicating the inverse function of the logarithm to the polynomial approximation to obtain an approximation of the histogram; determining a local maximum of the approximation of the histogram corresponding to healthy brain tissue and dividing the grey levels of the voxels of the three-dimensional image acquired according to the modality by the grey level corresponding to the local maximum in the histogram. (Kobald: page 106-107, section 4.2.4 Lesion Enhancement, As explained in section 1.2.2 the lesion can be seen on DWI but is easily confused with a couple of imaging artefacts. ADC tries to remedy this but has the disadvantage that it is a very crude approximation of the diffusion coefficient. The directional dependency of the diffusion coefficient and local imaging artefacts are disregarded by ADC and can render small lesions invisible (compare figure 4.4 b)). Remember that the ADC is calculated from the logarithm of the MRI signal)
Consider Claim 5.
Kobald teaches:
5. (Currently Amended) The method according to claim 3, including a processing step comprising the following sub-steps: for each imageimages; (Kobald: page 50
PNG
media_image4.png
300
664
media_image4.png
Greyscale
PNG
media_image5.png
690
672
media_image5.png
Greyscale
)
5. for each image of the set of images associated with the first modality, concatenating the image considered, the corresponding image of the set of images associated with the second modality and of the corresponding image of the set of images associated with the third modality, to obtain a set of concatenated second images. (Kobald: page 68 Figure 2.7, The principle of the bidirectional RNN illustrated for a bidirectional CLSTM working on a 3D volume. The forward model (left side) processes the volume slice by slice until the target slice. The backward model (right side) starts from the top and processes the slices in reverse order until the target slice. The output of both models for the target slice is concatenated (middle) and then passed through a classification layer to produce the prediction. The forward network sees only the data from the bottom to the target slice and the backward network sees only the data from the top to the target slice. The concatenated network outputs then contains information about the whole data volume and thus allows predictions which take the whole volume into account.
PNG
media_image6.png
248
700
media_image6.png
Greyscale
page 67 section 2.5.3 Backpropagation Through Time Recurrent neural networks (RNNs) can be trained by backpropagation through time. For this the network is unfolded, which means the loops in the network are turned into feed forward connections by replicating the networks structure. An unfolded LSTM is shown in figure 2.6, row one. After unfolding the network is represented as a directed acyclic graph where the original RNN is repeated over and over. In this unfolded representation most RNNs are very deep networks, which makes it clear why RNNs are counted as deep learning methods. But in contrast to a feed forward network many layers share their weights. The unfolded network is trained with backpropagation (i.e. gradient descent, section 2.2.2), as if every layer had its own set of weights. The final gradient for a parameter is the sum of all individual gradients which where calculated for that parameter throughout the unfolded network. For example if a parameter w appears m times in the unfolded network it will receive m different gradients ∆t. The gradient for w is then
PNG
media_image7.png
28
68
media_image7.png
Greyscale
)
Consider Claim 2.
Kobald teaches: 6. (Currently Amended) The method according to claim 5, wherein: supervised training of at least one primary recurrent artificial neural network is performed for a first primary recurrent artificial neural network and a second primary recurrent artificial neural network associated with different sets of training parameters, on a same primary database including a plurality of concatenated first images; the using of each primary recurrent artificial neural network trained is performed on a same primary set of images for the first primary recurrent artificial neural network and the second primary recurrent artificial neural network, the primary set of images corresponding to the set of concatenated first images; the supervised training of at least one secondary recurrent artificial neural network is performed for a first secondary recurrent artificial neural network and a second secondary recurrent artificial neural network having identical sets of training parameters, on a first secondary database for the first secondary recurrent artificial neural network and on a second secondary database for the second secondary recurrent artificial neural network, the first secondary database including a plurality of images acquired according to the first modality and the second secondary database including a plurality of concatenated second images;- the using of each secondary recurrent artificial neural network trained is performed on a first secondary set of images for the first secondary recurrent artificial neural network and on a second secondary set of images for the second secondary recurrent artificial neural network, the first secondary set of images corresponding to the set of images associated with the first modality and the second secondary set of images corresponding to the set of concatenated second images. (Kobald: page 67 section 2.5.3 Backpropagation Through Time Recurrent neural networks (RNNs) can be trained by backpropagation through time. For this the network is unfolded, which means the loops in the network are turned into feed forward connections by replicating the networks structure. An unfolded LSTM is shown in figure 2.6, row one. After unfolding the network is represented as a directed acyclic graph where the original RNN is repeated over and over. In this unfolded representation most RNNs are very deep networks, which makes it clear why RNNs are counted as deep learning methods. But in contrast to a feed forward network many layers share their weights. The unfolded network is trained with backpropagation (i.e. gradient descent, section 2.2.2), as if every layer had its own set of weights. The final gradient for a parameter is the sum of all individual gradients which were calculated for that parameter throughout the unfolded network. For example if a parameter w appears m times in the unfolded network it will receive m different gradients ∆t. The gradient for w is then
PNG
media_image7.png
28
68
media_image7.png
Greyscale
.)
Consider Claim 7.
Kobald teaches: 7. (Currently Amended) The method according to claim 1, wherein each primary recurrent artificial neural network and each secondary recurrent artificial neural network have the same architecture of the convolutional short- and long-term memory recurrent artificial neural network type having a memory, wherein each convolution is replaced with a logic block in which a first part of an input piece of data passes through a convolutional layer and a second part of the input piece of data passes through a transfer block comprised of a transfer layer surrounded by two convolutional layers, the transfer layer performing several max-pooling operations with different window sizes. (Kobald: pages 75-76 section 3.1.2 Transfer Block Definition, The proposed ”transfer block“ is a new building block for neural networks architectures. It provides an alternative to pyramidal feature extractors and uses significantly less parameters. Thus the “transfer block“ allows to build more lightweight models compared to contemporary models. The Transfer Block consists of three layers. The core part is the transfer layer which is enclosed by two conventional convolutional layers. This transfer layer builds upon the idea that information needs to be transferred across the image in order to have a maximum receptive field for the final neuron. This is achieved through multiple maximum pooling operations [74] with different window sizes. Be maxpool((Ii1i2i3)i3=t, B) the maximum pooling operation from equation 2.14 but restricted to the feature map t of the 2D image I with window size B and stride 1. Suppose a list of window sizes L where L(j) is the window size number j with a total number of nw window sizes. Be the multiplicity m an integer number and suppose that the number of channels of the input imagen3 = mnw. The input to the transfer layer is an image I of size n1 × n2 × n3 and the output is of the same size. On each of the input channels a single maximum pooling operation is performed and the result is the output for the corresponding channel. Each available window size in L is used on m distinct channels. To formalise this define an index u = jm + k with k ∈ {1, . . . , m}
PNG
media_image8.png
524
658
media_image8.png
Greyscale
)
Consider Claim 8.
Kobald teaches: 8. (Currently Amended) The method according to claim 7, wherein the training of a primary recurrent artificial neural network includes the following substeps for each image of the corresponding primary database: first submission of the image to the primary recurrent artificial neural network to fill its memory; second submission of the image to the primary recurrent neural network to provide lesion prediction from the memory filled, the second submission being immediately consecutive to the first submission; resetting the memory; and the step of training a secondary recurrent artificial neural network includes the following sub-steps for each image of the corresponding secondary database: first submission of the image consecutive to the first submission; and resetting(Kobald: pages 87-89, section 3.2 Logic LSTM, All segmentation architectures based on CLSTM use multiple CLSTMs (see section 2.5.5). This is due to inherent problems in the CLSTM itself. Its memory ct allows it to store information from the past, but for the purpose of prediction it stays a single convolution layer. That means a single CLSTM has the same problems as a single convolutional layer, i.e. a small receptive field (see section 2.4.3. The authors of CLSTM based segmentation methods [97, 94, 96, 95] solved this problem with the same approach as for CNNs by stacking layers. Of course this blows up the number of parameters in the models. A new more elegant approach solves the receptive field problems of the CLSTM and leads to the logic LSTM. The latter is built on two new concepts, the double pass and the logic block. section 3.2.1 Double Pass, Developed from the idea of the delayed input (see section 2.5.4) the double pass allows to incorporate the future as well the past of a finite known signal. Basically the model sees the signal twice with a delay equal to the signal length. First the entire signal is fed to the model, the memory pass, but the model’s output is ignored. So all that rests from the memory pass is what the model stores in its memory. Then the signal is fed a second time to the model, the prediction pass. This time the model’s output is recovered as prediction. This is visualised in figure 3.10. So for each time step (or slice in the case of MRI) in the signal the model produces a prediction using the current data item plus its memory. The memory contains information about the whole signal, i.e. past and future. Furthermore between the time step t in the memory pass and the time step t in the prediction pass exactly the signal length number of steps n passes. This allows the information of step t to be processed for n steps and as a result the prediction from a double pass model is from a deep network. Whereas in the bi-directional approach (section 2.5.4) the prediction for step t is immediate and thus from a shallow network. Also the bi-directional approach uses two distinct models where the double pass uses only one and thus saves 50% of the parameters. It will become apparent in the tests below that the double pass is indeed superior to a bi-directional approach. So compared to the bi-directional approach the double pass provides better information processing at the identical computational cost while using only half of the parameters.
PNG
media_image9.png
338
672
media_image9.png
Greyscale
)
Consider Claim 12.
Kobald teaches: 12. (Currently Amended) The method according to claim 1, comprising a step of volumetrically characterising the thrombus from the thrombus segmentation. (Kobald: pages 49-50, section 2.1.2 Working with Volumes, Volumes in the sense of this work are hypercubes of an N-dimensional space with a cartesian grid. Each position on the grid is attributed a value. Thus the volume is a finite set of values (voxels) v(i1, i2, . . . , iN ) : N N → R which are indexed through their position on the grid, i.e. by N indices i1, i2, . . . , iN with ik ∈ [1 . . . nk] . The nk are called the axis sizes of the volume. pages 87-88, section 3.2.1 Double Pass, Figure 3.10
PNG
media_image2.png
424
670
media_image2.png
Greyscale
pages 115-119, section 4.3.2. Mask R-CNN, section 4.3.3 Logic LSTM )
Consider Claim 14.
Kobald teaches: 14. (Currently Amended) A computer program product comprising instructions which, when the program is executed on a computer, cause the same to implement the steps of the methodclaim 1. (Kobald: pages 48-49 Chapter 2 Machine learning Machine learning is, in the widest sense, building machines which are able to learn a task by observation of examples. Currently these machines are computer programs and the algorithms are closely linked to statistics and aimed at modelling probability densities which describe some data. Any of the current models obey the same principle.
PNG
media_image1.png
378
668
media_image1.png
Greyscale
page 60 section 2.4.2. Neural Networks, section Recurrent Networks, In contrast to the feed forward network a recurrent neural network (RNN) can be a cyclic graph. A step can be dependent on the result of a future step, or even its own result. Also network models with a memory (i.e. ~y depends not only on ~x but also on all other inputs the model has seen in the past or future) fall under the definition of a recurrent network. This comes from the fact that ”memory“ can be represented by a self loop in the model. Both types are constructed as a sequence of operations, also called layers, with simple to calculate gradients. This structure makes training with gradient descent (section 2.2.2) quite easy. Via the chain rule for derivatives the derivative of the loss can be calculated analytically for each layer. This results in the gradient of each layer being dependent on the derivatives of the following layers. This can be seen as the error being propagated ”backwards“ through the network in order to calculate the gradients of the individual layers. Hence the name ”backpropagation“ for training of neural networks with gradient descent. There are many choices for layers. The most popular ones are dense layers (equation 2.31), convolution layers (equation 2.13) and maximum pooling layers (equation 2.14).)
Consider Claim 15.
Kobald teaches: 15. (Currently Amended) A non-transitory computer-readable recording medium comprising instructions which, when executed by a computer, cause the same to implement the steps of the method according to claim 1. (Kobald: pages 48-49 Chapter 2 Machine learning Machine learning is, in the widest sense, building machines which are able to learn a task by observation of examples. Currently these machines are computer programs and the algorithms are closely linked to statistics and aimed at modelling probability densities which describe some data. Any of the current models obey the same principle.
PNG
media_image1.png
378
668
media_image1.png
Greyscale
page 60 section 2.4.2. Neural Networks, section Recurrent Networks, In contrast to the feed forward network a recurrent neural network (RNN) can be a cyclic graph. A step can be dependent on the result of a future step, or even its own result. Also network models with a memory (i.e. ~y depends not only on ~x but also on all other inputs the model has seen in the past or future) fall under the definition of a recurrent network. This comes from the fact that ”memory“ can be represented by a self loop in the model. Both types are constructed as a sequence of operations, also called layers, with simple to calculate gradients. This structure makes training with gradient descent (section 2.2.2) quite easy. Via the chain rule for derivatives the derivative of the loss can be calculated analytically for each layer. This results in the gradient of each layer being dependent on the derivatives of the following layers. This can be seen as the error being propagated ”backwards“ through the network in order to calculate the gradients of the individual layers. Hence the name ”backpropagation“ for training of neural networks with gradient descent. There are many choices for layers. The most popular ones are dense layers (equation 2.31), convolution layers (equation 2.13) and maximum pooling layers (equation 2.14).)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kobald et al. (Jonathan Kobold. Deep Learning for lesion and thrombus segmentation from cerebral MRI. Image Processing [eess.IV]. Université Paris Saclay (COmUE), 2019. English. ⟨NNT : 2019SACLE044⟩. ⟨tel-03592570⟩ Available online: https://theses.hal.science/tel-03592570v1, hereby referred to as “Kobald”), in view of Brynolfsson et al. (US PGPub US 20230351586 A1), hereby referred to as “Brynolfsson”. Kobald was cited by applicant in IDS submitted on .January 4, 2024.
Consider Claim 9.
Kobald teaches:
9. (Currently Amended) The methodclaim 1, wherein the thrombus segmentationmethod comprising a step of refining the thrombus segmentation comprising the following sub-steps of: calculating the lesion envelope from the lesion segmentation, distributing the thrombus segmentation into a first set of voxels and a second set of voxels: the first set of voxels includes the voxels of the thrombus segmentation associated with the false-positive label, the second set of voxels includes the voxels of the lesion segmentation associated with the true-positive label, reducing the first set of voxels of the thrombus segmentation: for each voxel of the first set of voxels, calculating the distance to the lesion, the distance to the lesion being the (Kabold: page 37 section 1.2.6. Diagnosis Pipeline The diagnosis pipeline is as follows: FLAIR: – Provides the size of the core and in comparison with the cytotoxic oedema from DWI an estimate of the penumbra. – Allows to distinguish the lesion from the current stroke and old lesions from previous incidents. – Rough size of the penumbra can be estimated from collateral arteries. Page 47 section 1.3.6 RAPID RApid processing of PerfusIon and Diffusion (RAPID,[72]) is the only automated feature extraction pipeline which was developed for clinical use. It uses perfusion and diffusion MRI. From the perfusion MRI the time Tmax from contrast bolus application to maximum intensity is calculated for each voxel. From the DWI the ADC map is calculated. Both the Tmax and the ADC map are thresholded with a patient independent threshold. After removal of small components the two binary masks are compared. The mismatch between the two is the perfusion diffusion mismatch, an indicator for the penumbra.
9. ordering the voxels of the first set of voxels as a function of the distance to the lesion calculated, selecting a subset of voxels from the ordering of the voxels in the first set of voxels, selecting the segmentation corresponding to the merging of the second set of voxels and the subset of voxels selected as the thrombus segmentation. (Kabold: pages 100-101, section 4.1.1 Sampling, Sampling in this context is the creation of sub-images from an MRI for the use in a machine learning method. This is necessary because the whole MRI of a patient is too large to be processed in one piece. This in turn means that the machine learning models don’t see the whole MRI. The sub-images, called patches, have to show relevant parts of the brain in order to produce high-performance models. For example a thrombus has an median size of 542 voxels. If patches are randomly sampled from a SWAN with roughly 40 million voxels the chance is high that the thrombus never appears on a single patch. Because of this class imbalance the models trained on these patches can never learn what a thrombus looks like. Other problems which arise are border problems, where thrombi situated at the edges of a patch have not enough context to be correctly identified. Then there are also objects which look very much like the thrombus and are likely to cause false positives. These “likely false positive” objects are also rare with respect to the millions of voxels in an MRI. To counter all these problems a specific sampling strategy has been chosen which ensures that the training sets contain relevant data. Border problems are avoided by an overlapping patches approach as proposed by Ronneberger et al. [87]. That means that the border of a patch is ignored and only the predictions of the centre, where enough context is available, are used. As the predictions are only for the centre of the patches it is safe to create the patches in a way that the relevant information is centred. The class imbalance is eliminated by choosing an equal number of patches with thrombus and “likely false positive” objects. During training it is assured that every second image in a mini-batch shows a thrombus and every other shows a “likely false positive” object. This forces the models to concentrate on the differences between thrombus and “likely false positive” objects. Easier to learn non-thrombus tissues are learned alongside as non-thrombus tissues are always on the patches surrounding the thrombus. The patch size has to be chosen small enough that a thrombus makes up a relevant amount of voxels in the patch (at least 1%) and large enough that the surroundings are visible. “Likely false positive” objects are defined as areas with the same intensity range as the thrombus inside the brain. This sampling method applies to the lesion as well.)
Kobald does not necessarily teach: the distance to the lesion being the Euclidean distance from the voxel in the first set of voxels to the lesion envelope,
Brynolfsson teaches:
1. (Currently Amended) An automatic method(Brynolfsson: abstract, Presented herein are systems and methods that provide for improved detection and characterization of lesions within a subject via automated analysis of nuclear medicine images, such as positron emission tomography (PET) and single photon emission computed tomography (SPECT) images. In particular, in certain embodiments, the approaches described herein leverage artificial intelligence (AI) to detect regions of 3D nuclear medicine images corresponding to hotspots that represent potential cancerous lesions in the subject. The machine learning modules may be used not only to detect presence and locations of such regions within an image, but also to segment the region corresponding to the lesion and/or classify such hotspots based on the likelihood that they are indicative of a true, underlying cancerous lesion. This AI-based lesion detection, segmentation, and classification can provide a basis for further characterization of lesions, overall tumor burden, and estimation of disease severity and risk.)
9. ordering the voxels of the first set of voxels as a function of the distance to the lesion calculated, (Brynolfsson: [0232] FIG. 3 shows an example process 300 for correcting intensity blead from a high-uptake tissue region. As shown in FIG. 3, a 3D functional image is received 304 and a high intensity volume corresponding to the high-uptake tissue region is identified 306. In another step, a suppression volume outside the high-intensity volume is identified 308. In certain embodiments, as described herein, the suppression volume may be determined as a volume enclosing regions outside of, but within a pre-determined distance from, the high-intensity volume.)
9. for each voxel of the first set of voxels, calculating the distance to the lesion, the distance to the lesion being the Euclidean distance from the voxel in the first set of voxels to the lesion envelope, (Brynolfsson: [0412] The adjustment corrects for small misalignments between the PET and CT images. Using the adjusted map, a background image is calculated. This background image is subtracted from the original PET image and creates an uptake estimation image. The suppression map is then estimated from the uptake estimation image using an exponential function that is dependent on a Euclidean distance from a voxel outside the segmentation to the PET adjusted organ mask. An exponential function is used since the uptake intensity decreases exponentially with distance from the organ. Finally, the suppression map is subtracted from the original PET image, thereby suppressing intensities associated with high normal uptake in the organ.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Kobald’s method and system for Deep learning for thrombus and lesion segmentation to leverage known and established distance metrics as disclosed by Brynolfsson for AI-based characteristics of lesions. The determination of obviousness is predicated upon the following findings: Both are directed towards the overall field of machine learning and AI algorithms for lesion detection and medical diagnosis and One skilled in the art would have been motivated to modify Kobald’s distance parameters in order to use established metrics such as Euclidean distances for lesion analysis as suggested by Brynolffson. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kobald, while the teaching of Bryonlfsson’s use of Euclidean distance metrics would have been a predictable modification that would yield a reasonable expectation of success, and would continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of ensuring accuracy in identifying and segmentation of lesions and thrombi in medical imaging and analysis. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 10.
The combination of Kobald and Brynolfsson teaches: 10. (Currently Amended) The method according to claim 9,wherein the step of selecting a subset of voxels is performed by selecting the N voxels of the first set of voxels having the smallest distance to the lesion, N being between 3 and 5. (Kobald: page 45 The first term is the average intensity distance between the intensity x of a point and the intensities ni , i ∈ {1, . . . , k} of its k nearest neighbours . The second term is the mean intensity distance between the nearest neighbours. ζ is a measure of anomaly of a voxel with respect to its nearest neighbours. A cube of 3×3×3 voxels is used as the neighbourhood. ζ is used in conjunction with an atlas of normal ζ values. Brynolfsson: [0412] The adjustment corrects for small misalignments between the PET and CT images. Using the adjusted map, a background image is calculated. This background image is subtracted from the original PET image and creates an uptake estimation image. The suppression map is then estimated from the uptake estimation image using an exponential function that is dependent on a Euclidean distance from a voxel outside the segmentation to the PET adjusted organ mask. An exponential function is used since the uptake intensity decreases exponentially with distance from the organ. Finally, the suppression map is subtracted from the original PET image, thereby suppressing intensities associated with high normal uptake in the organ.)
Consider Claim 11.The combination of Kobald and Brynolfsson teaches: 11. (Currently Amended) The method according to claim 9, wherein the step of selecting a subset of voxels is performed by selecting the voxels of the first set of voxels having a distance to the lesion less than a predetermined threshold. (Kabold: page 113 The three models provide three different scores s1, s2, s3 for each voxel (figure 4.7 d)-f)) which need to be merged for a final decision. A simple majority vote would underestimate the size of the thrombus. Thus the label merging is split into two parts. At first possible thrombi are extracted and in the second part these candidates are classified into thrombus and non thrombus. For the first part a new volume V is created where each voxel is assigned the maximum max(s1, s2, s3) out of the three scores. V is then thresholded at a value of 0.4 and the resulting binary map is divided into connected components. To decide whether one component should be a thrombus or not, the scores (s1, s2, s3) are used again. If all three scores are bigger than 0.7 for one voxel in the component, i.e. s1 > 0.7 ∧ s2 > 0.7 ∧ s3 > 0.7, then the component is validated as a thrombus (figure 4.7 c)). Brynolfsson: [0096] In certain embodiments, the analytical model is an adaptive thresholding method, and step (i) comprises: determining one or more reference values, each based on a measure of intensities of voxels of the 3D functional image located within a particular reference volume corresponding to a particular reference tissue region. [0097] In certain embodiments, the hotspot-specific threshold value is determined using a particular threshold function selected from a plurality of threshold functions, the particular threshold function selected based a comparison of the corresponding hotspot intensity with the at least one reference value [e.g., wherein each of the plurality of threshold functions is associated with a particular range of intensity (e.g., SUV) values, and the particular threshold function is selected according to the particular range that the hotspot intensity and/or a (e.g., predetermined) percentage thereof falls within (e.g., and wherein each particular range of intensity values is bounded at least in part by a multiple of the at least one reference value)].)
Allowable Subject Matter
Claims 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 13 is not rejected because the prior art fails to teach the method of Claim 13, which specifically comprises the following features in combination with other recited limitations:
-; 13. (Currently Amended) The method 12,wherein the step of volumetrically characterising the thrombus comprises the following sub- steps of: extracting the pixels of the thrombus segmentation from the thrombus segmentation, the pixels forming a point cloud contained in a volume, the volume being the maximum ellipsoidal envelope containing the thrombus, determining the geometric characteristics of the volume of the maximum ellipsoidal envelope containing the thrombus by principal component analysis of the point cloud along three main axes, the three main axes being orthogonal to each other, calculating the volume of the maximum ellipsoidal envelope containing the thrombus from the geometric parameters determined.
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image10.png
60
904
media_image10.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
December 11, 2025
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674