DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 11, filed 1/22/2026, with respect to claim objections have been fully considered and are persuasive. The objections of the claims has been withdrawn. The objection to the specification is maintained.
Applicant's arguments filed 1/22/2026 have been fully considered but they are not persuasive. The Applicant’s remarks in the specification state that the combined references do not disclose “inputting the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image;
inputting the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images (150) arising from corresponding motion of the vasculature between successive contrast- enhanced images in the temporal sequence”. The Examiner respectfully disagrees with this assertion based on the reference of Crabb and will briefly explain why below.
Regarding the Crabb reference, this system discloses a PatchGan that includes a generative neural network (i.e. generator) and a discriminative neural network (discriminator). The generator receives images, or patches, that contain motion artifacts in dataset C. The generator outputs the image patches without the motion artifacts. The patches output can represent temporal motion-compensated DSA images, and if continually recorded as stated in ¶ [31], a sequence of images can be considered in the dataset. The primary reference discloses a temporal sequence of images in ¶ [12]-[16] and in figure 4. This combination discloses evaluating a patch of images, whether of a temporal sequence of DSA images of a particular vascular image or a single DSA image of different vascular areas (i.e. ¶ [34]-[37] in Crabb), in order to compensate for the motion within the input DSA images. With the compensation of motion occurring with a first neural network of images that are input in order to output these same images in a motion compensated manner, the first contended limitation is performed.
Regarding the second limitation, the discriminative model is the second neural network that receives the motion-compensated DSA images from the generative model. The discriminator in the PatchGan process discriminates whether or not each patch is real or fake. As the results are output, the additional patches are also evaluated with the real or fake result averaged with the past results. The discriminator groups the patches together into an overall image and provides a result of the overall image. This performs the aspect of receiving multiple inputs, creating a composite image and predicting the overall image to determine whether the motion compensated image is real or fake. These two neural networks are outlined in ¶ [37]-[42] and [51]-[54]. With the PixelGan operation including a discriminator and generator able to perform the above features, the features of the claims are disclosed.
Thus, based on the above, the rejection of the claims are disclosed below.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested:
MOTION COMPENSATION IN ANGIOGRAPHIC IMAGES THROUGH TRAINING A GENERATIVE ADVERSARIAL NETWORK TO PREDICT THE COMPOSITE MOTION COMPENSATED DIGIAL SUBTRACTION ANGIOGRAPHIC MAGES.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5 and 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rauch (US Pub 2012/0201439) in view of Crabb (US Pub 2022/0409161 (PCT Filing date: 12/1/2020)).
AMENDMENTS TO THE CLAIM
1. (Currently amended) Rauch discloses a computer-implemented method of performing motion compensation on a temporal sequence of digital subtraction angiography,(DSA); the method comprising:
receiving (S110) a temporal sequence of DSA images (110) of a vasculature generated by subtracting a mask image from a temporal sequence of contrast-enhanced images (e.g. images of DSA image sequence is generated based on subtracting a mask image from a shifted image to produce a subtracted image enhancing vessel structure, which is taught in ¶ [14], [19] and [20].);
[0014] Interface 23 acquires a sequence of images of patient vessels both prior to and following introduction of contrast agent into the vessels. Image data processor 15 automatically, determines a first shift vector for a first image of the sequence for compensating for shift between the first image and a first reference image (e.g., a mask image) of the sequence and applies the determined first shift vector to the first image of the sequence to produce a shifted image. Processor 15 subtracts the first reference image from the shifted image to produce a subtracted image enhancing vessel structure. Further, processor 15 determines a second shift vector for compensating for shift between the subtracted image and a second reference image (e.g., a contrast entrance image) and shifts content of the subtracted image relative to the second reference image in response to the second shift vector, to provide a shifted subtracted image enhancing and aligning vessel structure.
[0019] Image data processor 15 automatically applies the determined first shift vectors to corresponding multiple images of the sequence to produce multiple shifted images 404-410. Processor 15 automatically shifts images in the image sequence (or at least images following contrast entrance), using a fixed shift vector that when applied to an individual selected image minimizes the difference between the individual selected image and mask image 402. Processor 15 subtracts the mask (first reference) image from the multiple shifted images to produce subtracted images comprising a digitally subtracted Angiography (DSA) sequence enhancing vessel structure. Thereby processor 15 derives individual images of a standard DSA image sequence by applying a shift vector of each image to that image and subtracting the mask image.
[0020] Processor 15 identifies contrast enhanced images (corresponding to 404, 406, 408, and 410) in the image sequence and the first image (404) in the sequence first exhibiting presence of contrast agent (a contrast agent entrance image). Processor 15 identifies contrast agent enhanced images in an image sequence, by in one embodiment, for individual images of a sequence, deriving a histogram as a measure representative of luminance content of each individual image. Processor 15 compares differences between the histogram measures of successive images. In the histogram, a horizontal axis represents each possible luminance pixel value from black to white. The vertical axis indicates values representing the number of pixels in the image that occur at each luminance pixel value level. Processor 15 generates and analyzes histograms for a ROI (or the whole image) of individual images. Processor 15 processes the pixel data within a determined ROI to derive the ROI pixel luminance (e.g., grayscale) distribution comprising a histogram. Processor 15 compares and correlates histogram data representing successive images of a medical image sequence of a patient anatomical portion to identify a first image of a sequence in which change of luminance data (e.g., increased darkness representing Iodine contrast agent) occurs in the sequence to determine when the contrast agent enters the ROI and to identify images containing contrast agent. In response to determining a difference in measures representative of luminance content (histograms) of an image exceeding a predetermined threshold. Image processor 15 selects an image comprising the image immediately preceding the contrast agent entrance image as a Mask image. Processor 15 also initiates image comparison after administration of the contrast agent to ensure that there is sufficient background image variation for acquiring histogram data for comparison and correlation.
temporal sequence of motion-compensated DSA images (150) arising from corresponding motion of the vasculature between successive contrast- enhanced images in the temporal sequence (e.g. the temporal sequence of motion-compensated DSA images come from the vasculature between successive contrast-enhanced images in the temporal sequence, which is taught in ¶ [21]-[25].),
predict, based on motion artifacts in the DSA images, a composite motion-compensated DSA image representing the temporal sequence of the DSA image and including (i) compensation for motion of the vasculature between successive contrast- enhanced images in the temporal sequence; and (ii) compensation for motion of the vasculature between acquisition of the contrast-enhanced images in the temporal sequence and acquisition of the mask image (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25]. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25].); and
[0012] A system according to invention principles employs a multi-level pixel-shift to compensate for an additional motion field associated with contrast enhanced image structures. The system aligns content of individual images of a multi-image sequence to enable improved image to image correlation and multiple image composition. This is accomplished by removing background motion and foreground motion. In this case foreground motion comprises motion applied to an object of interest that is different than the motion applied to image background detail. The system performs a multi-level pixel-shift to align content of multiple images of a pixel-shifted DSA image and to allow improved multiple image processing and compositional review. The system improves pixel-shift motion correction in DSA image processing to align the content of individual images of a multi-image sequence with a specific mask image to reduce background motion and to align resulting subtracted images to reduce secondary motion applied to the area of anatomy being imaged.
[0021] Image data processor 15 automatically (or in response to user command) identifies a ROI in the contrast enhanced structures (foreground) of images corresponding to images 404-410. Automatic selection of the ROI is performed by processor 15 based on image acquisition program settings, data identifying a clinical procedure, or image analysis. Image analysis in one embodiment is used to detect a fixed size ROI that contains the highest concentration of contrast enhanced pixels to select a ROI that encompasses a large portion of the contrast enhanced structure. Automatic selection of the ROI is also performed using image analysis to detect specific anatomy (e.g. spine, ribs, skull, or other bony structure) associated with a ROI and the detected anatomy is used to determine the ROI comprising a predetermined area associated with the detected particular anatomical features. The system selects a ROI as comprising static background image content that does not vary (or varies only slightly) between images in a sequence.
[0022] Processor 15 automatically shifts contrast agent enhanced images (following the first contrast entrance image) in the DSA image sequence. Image data processor 15 determines multiple second shift vectors for corresponding multiple images of the DSA sequence exhibiting presence of contrast agent for compensating for shift between individual images of the images of the DSA sequence and a contrast agent entrance image of the DSA sequence. In one embodiment processor 15 determines multiple second shift vectors for compensating for shift between a subtracted image and an image immediately preceding the subtracted image. The pixel-shift applied to each image is a pixel-shift relative to the contrast entrance image. In another embodiment, processor 15 determines multiple second shift vectors for compensating for shift between individual subtracted images of the DSA sequence and a contrast agent entrance image of the DSA sequence.
[0023] Processor 15 shifts content of the individual images of the multiple images of the DSA sequence relative to the contrast entrance image in response to the second shift vector, to provide a shifted DSA sequence enhancing and aligning vessel structure. This pixel-shift is advantageously different to conventional pixel-shift calculations in that shift vectors are determined to be relative to a contrast entrance image. The shift vector for an individual image determines the shift vector that aligns the individual image to the pixel coordinate space of the contrast entrance image. This pixel-shift can either be determined for a given image by comparing the image against the contrast entrance image or against the image immediately preceding the given image. In one embodiment the system uses the mask image instead of the contrast entrance image if there is sufficient similarity between the contrast entrance image and contrast enhanced images (as is the case in a continuous contrast injection into a vessel, for example). Short intra-arterial contrast injections do not have similar content between the contrast entrance image and the subsequent images because the contrast bolus flows through the vasculature and is absent in later images from the arteries in which the contrast agent is present in earlier images.
[0024] A resulting image 412 shows the vessels are aligned following superimposing images 404-410 and a similar composite flow image may be generated in different colors (represented by gray shades in image 414) that shows progression of blood flow through the vessel structure upon contrast agent injection. Image 416 shows both vessels 425 and bones 415 are aligned.
[0025] The position of the foreground ROI is variable and determined for each image and processor 15 iteratively removes the effects of more than two types of motion. The system in one embodiment adaptively selects a pixel-shift function to avoiding need to select a background ROI. Quality measurements of individual pixel-shift results are used to determine if a calculated shift vector is valid or to adjust or interpolate a shift vector to find a better one. The system is applicable to multiple images of a sequence in which there are multiple objects in motion (for example, security imaging, scientific image analysis (e.g. microscopy, astronomy)). The system advantageously uses multiple levels of pixel-shift to compensate for multiple types of motion occurring in content of an imaged area and is used to process multiple images or to combine multiple images for display. The system automatically determines positions of multiple regions of interest associated with different corresponding motion vectors in an image or in another embodiment a user manually identifies and positions these regions of interest. The system performs the multi-level pixel-shift and the processing, superimposition or combination of multiple images.
outputting the predicted composite motion-compensated DSA image (e.g. the determined composite motion-compensated DSA image can be output to a display, which is taught in ¶ [29] and [30].).
[0029] A processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and is conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.
[0030] An executable application, as used herein, comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters. A user interface (UI), as used herein, comprises one or more display images, generated by a user interface processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions.
However, Rauch fails to specifically teach the features of inputting the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses inputting the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding
[0030] The inventors have performed extensive experiments on real medical images to prove that the principles described herein produce the result as intended. For example, the inventors used DSA images as training and testing data to train a generator G and a discriminator D. The trained generator G not only can effectively enhance DSA images that have significant motion artifacts, but also enhance post-contrast images prior to subtraction of the mask image, eliminating the step of subtraction of mask image.
[0037] In the example embodiment, dataset C consisted of visceral DSA images with significant motion artifacts. Patients were identified by searching mPower for the phrase ‘IR Visceral Angio’. Imaging studies for patients meeting this criterion were completely anonymized and exported as DICOM files for the inclusion in our research. Twenty-two individual patients (14 males, 8 females) were identified that met this criterion, for a total of 130 series and 3,322 DSA images. Patients in the dataset have a median age of 62 years [range 40 -79 years] and a median body mass index (BMI) of 28.32 kg/m.sup.2 [range 14.84 -40.45 kg/m.sup.2].
[0038] FIG. 1 illustrates native fluoroscopic image patches with injected contrast (512×512 pixels) used as inputs to the neural network during training and the corresponding DSA patches used as ground-truth outputs during training. The datasets A and B consist of training image pairs, with the original native fluoroscopic image with injected contrast as the input and the DSA without motion artifacts as a ground truth output. An example of these training pairs can be seen in FIG. 1. Datasets A and B were randomly split into training, testing, and validation datasets using an 80-10-10 percent split. During this split, images from the same x-ray series were confined to the same dataset. Additionally, the original images, which were frequently 1024×1024 pixels or larger, were split into 512×512 pixel patches. As a result, dataset A produced a total of 23,784 training images, 2,912 validation images, and 2,960 testing images. Dataset B produced a total of 5,065 training images, 503 validation images, and 775 testing images.
[0039] Conditional adversarial networks excel on large datasets of aligned image pairs, such as the input and output pairs in datasets A and B. These architectures consist of two distinct neural networks: a generator (G) and discriminator (D). As shown in FIG. 2, the generator is used to generate the output images (X.sub.g) from the input (Z). The discriminator attempts to discern real images (X.sub.r) in the training data from artificially generated images (X.sub.g), producing a classification of real or fake (Y.sub.1). Using this process, the generator progressively produces more accurate and realistic images as the discriminator gets more adept at distinguishing between artificial and real DSA images.
[0040] In at least one embodiment, the disclosed system utilizes the pix2pix architecture as a conditional GAN. The pix2pix architecture further utilizes the neural network Unet as the generator. Unet is an encoder-decoder with skip connections between mirrored layers in the encoder and decoder stacks. As such, in at least one embodiment, the system uses the Unet as the generator for the network; however, one will appreciate that several different architectures may be used for the discriminator network.
[0041] The performance of Unet in combination with three different discriminator architectures was investigated. The primary discriminator architecture was part of the original pix2pix implementation. This discriminator focuses on local image patches and attempts to classify each N×N patch in an image as real or fake. The algorithm was then run convolutionally across the image, averaging responses to produce an ultimate classification of real or fake. As a result, this discriminator assumes independence between pixels separated by Nand effectively models the image as a Markov random field.
[0051] When performing the qualitative assessment, a visual review of generated images by the proposed methods for the cerebral and hepatic angiograms (datasets A and B) can be seen in FIG. 5. FIG. 5 illustrates image results for each algorithm combination trained on datasets A and B, the cerebral and hepatic angiograms respectively. The first column shows the fluoroscopic images used as inputs, the second column shows ground truth DSA images, and the third through sixth columns show outputs generated using each network architecture. For both the cerebral and hepatic angiograms, each network is able to accurately identify the vasculature of interest and reproduce the ground truth DSA images in the setting of no motion artifacts. However, the method Unet+PatchGAN shows the most consistent ability to sharply delineate the vasculature of interest.
[0052] Additionally, to assess the ability of the networks to reliably eliminate motion artifacts, a visual review of images generated from dataset C, the visceral angiograms containing significant motion artifacts, was performed. Representative examples from this visual review are shown in FIG. 6. FIG. 6 illustrates image results for the fully trained algorithm combinations tested on dataset C, which contains visceral angiograms with significant motion artifacts. The first column shows the fluoroscopic images used as inputs, the second column shows traditional DSA images, and the third through sixth columns show outputs generated using each network architecture. The generated images show successful suppression of motion artifacts that were present when using the traditional DSA method; furthermore, the method Unet+PatchGAN shows the most consistent ability to accurately identify the vasculature of interest.
[0053] Additionally, a visual comparison of images generated from dataset C for the method Unet+PatchGAN, trained with and without transfer learning on dataset A (cerebral angiograms), can be seen in FIG. 7. FIG. 7 illustrates a qualitative comparison of Unet+PatchGAN performance on visceral angiograms with motion artifacts (dataset C), with and without transfer learning from cerebral angiograms (dataset A). Transfer learning from dataset A significantly improves the algorithm's ability to reliably eliminate motion artifacts from visceral angiograms and identify the vasculature of interest (solid arrows).
[0054] In at least one embodiment, DSA images generated using the Unet+PatchGAN architecture successfully suppress motion artifacts in visceral angiograms while also preserving critical structures such as the vasculature. By utilizing transfer learning, this technology can be applied to areas of the body, such as the viscera, where training data is critically limited. Furthermore, this motion correction is performed in near real time, with each 512×512 pixel image being processed in 0.023 seconds on average. In certain clinical circumstances, this approach already has the potential to aid in identifying the direction of a particular vessel that might otherwise be obscured by motion.
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of inputting the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image; inputting the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 2: (Currently amended) However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 1, wherein the second neural network is trained to predict, from the input of the temporal sequence, the composite motion-compensated DSA image, by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as not having motion artifacts, and a plurality of DSA images of the vasculature classified as having motion artifacts; and
inputting the DSA images of the vasculature classified as having motion artifacts from the DSA training image data, into the neural network, and adjusting parameters of the neural network based on a first loss function representing a difference between a composite motion-compensated DSA image predicted by the second neural network, and a combined image representing the inputted DSA training image data, and
based on a second loss function representing a probability of the composite motion-compensated DSA image predicted by the second neural network corresponding to a DSA image of the vasculature classified as not having motion artifacts from the DSA training image data.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein a neural network is trained to predict, from the input of the temporal sequence, the composite motion-compensated DSA image (e.g. a neural network is trained with the input of DSA images, in order to predict or output a DSA image without motion artifacts, which is taught in ¶ [38]-[40].), by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as not having motion artifacts, and a plurality of DSA images of the vasculature classified as having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37].); and
[0034] In an embodiment example, a system is trained to enhance DSA images. The system utilizes three datasets as training datasets that will be referred to as dataset A, B, and C respectively. For all three datasets in this example, imaging studies were generated using Artis angiography systems manufactured by Siemens Healthcare at the University of Utah Hospital or the Huntsman Cancer Institute and were identified and selected using Nuance mPower Clinical Analytics and the University of Utah's NUMAstore PACS system. For all datasets, the presence or absence of motion artifacts was assessed by a human expert.
[0035] In the example embodiment, dataset A consists of DSA images of the carotid and cerebral vasculature with minimal or no motion artifacts. Patients were identified by searching mPower for the phrase ‘Cerebral Angiogram “general anesthesia”’. Imaging studies for patients meeting this criterion were anonymized and exported as DICOM files for the inclusion in our research. Due to the rigidity of the skull and the administration of general anesthesia, these cerebral angiograms were considered to be the ideal representations of DSA images without motion artifacts. 35 patients (14 males, 21 females) were identified that met this criterion, for a total of 400 series and 7,958 DSA images. Patients in this dataset have a median age of 57 years [range 24 -77 years] and a median body mass index (BMI) of 28.38 kg/m.sup.2 [range 16.14 -43.86 kg/m.sup.2].
[0036] In the example embodiment, dataset B consists of DSA images of the hepatic vasculature with minimal or no motion artifacts. Patients were identified by searching mPower for the phrase ‘IR Visceral Angio “hepatic”’. Imaging studies for patients meeting this criterion were anonymized and exported as DICOM files for the inclusion in our research. Thirty-one patients (19 males, 12 females) were identified that met this criterion, for a total of 74 series and 1,203 DSA images. Patients in the dataset have a median age of 59 years [range 18 -77 years] and a median body mass index (BMI) of 26.30 kg/m.sup.2 [range 19.60 -52.80 kg/m.sup.2].
[0037] In the example embodiment, dataset C consisted of visceral DSA images with significant motion artifacts. Patients were identified by searching mPower for the phrase ‘IR Visceral Angio’. Imaging studies for patients meeting this criterion were completely anonymized and exported as DICOM files for the inclusion in our research. Twenty-two individual patients (14 males, 8 females) were identified that met this criterion, for a total of 130 series and 3,322 DSA images. Patients in the dataset have a median age of 62 years [range 40 -79 years] and a median body mass index (BMI) of 28.32 kg/m.sup.2 [range 14.84 -40.45 kg/m.sup.2].
inputting the DSA images of the vasculature classified as having motion artifacts from the DSA training image data, into the neural network, and adjusting parameters of the neural network based on a first loss function representing a difference between a composite motion-compensated DSA image predicted by the neural network, and a combined image representing the inputted DSA training image data (e.g. dataset A is input into the network in order to be used in training with a validation dataset. An L1 loss is calculated regarding the dataset A, which is taken into account in the objective function of the GAN. This is shown in ¶ [43].), and
[0043] In at least one embodiment, the objective function of a conditional GAN can be expressed as,
L.sub.CGAN(G,D)=E.sub.x,y[logD(x,y)]+E.sub.x,z[log(1−D(x,G(x,z))],
where the adversarial discriminator, D, tries to maximize the objective and the generator, G, tries to minimize it. This term is described as the adversarial loss; however, it is also paired with a more traditional L1 distance loss function defined as,
L.sub.L1(G)=E.sub.x,y,z[||Y−G(x, z)||.sub.1].
Consequently, the final objective function can be expressed as,
[00001]G*=argminGmaxDLcGAN(G,D)+λLL1(G),
where λ is a weight term applied to balance the adversarial and L1 losses. Unless otherwise specified, this objective function was used for all training.
[0045] Training was started on dataset A, using random weight initialization, and performed for a total of 30 epochs. Progress was tracked using an L1 loss on the validation dataset and a visual review of the generated DSA images. After training was complete, the algorithm was implemented on the testing dataset and results were manually reviewed (e.g., reviewed by an expert or a board-certified radiologist). Next, using the pretrained weights from dataset A, fine tuning was performed on dataset B for a total of 10 epochs. Progress was tracked using an L1 loss on the validation dataset and a visual review of the generated DSA images. Validation and training loss curves for training on datasets A (cerebral angiograms) and B (hepatic angiograms) can be seen in FIGS. 3 and 4 respectively. More specifically, FIG. 3 illustrates training and validation loss curves for each algorithm combination training on the cerebral DSA images (dataset A), and FIG. 4 illustrates training and validation loss curves for each algorithm combination training on the hepatic DSA images (dataset B).
based on a second loss function representing a probability of the composite motion-compensated DSA image predicted by the neural network corresponding to a DSA image of the vasculature classified as not having motion artifacts from the DSA training image data (e.g. the adversarial loss includes a loss that is used to output a loss value associated with the probability of the DSA image predicted being classified as a real image that is close to the training data without artifacts. This is taught in ¶ [43].).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein a neural network is trained to predict, from the input of the temporal sequence, the composite motion-compensated DSA image, by: receiving DSA training image data including a plurality of DSA images of the vasculature classified as not having motion artifacts, and a plurality of DSA images of the vasculature classified as having motion artifacts; and inputting the DSA images of the vasculature classified as having motion artifacts from the DSA training image data, into the neural network, and adjusting parameters of the neural network based on a first loss function representing a difference between a composite motion-compensated DSA image predicted by the neural network, and a combined image representing the inputted DSA training image data, and based on a second loss function representing a probability of the composite motion-compensated DSA image predicted by the neural network corresponding to a DSA image of the vasculature classified as not having motion artifacts from the DSA training image data, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 3: (Currently amended) Rauch discloses the computer-implemented method according to claim 1, wherein the composite motion-compensated DSA image is
a candidate composite motion-compensated DSA image representing the inputted temporal sequence of DSA images, and including compensation for the motion of the vasculature between successive contrast-enhanced images in the temporal sequence, and including compensation for motion of the vasculature between the acquisition of the contrast-enhanced images in the temporal sequence and the acquisition of the mask image (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25] above. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25] above.).
However, Rauch fails to specifically teach the features of wherein the composite motion-compensated DSA image is predicted by the second neural network that comprises:
a generative model trained to predict, from the inputted temporal sequence of DSA images, a candidate composite motion-compensated DSA image representing the inputted temporal sequence of DSA images
wherein the neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the composite motion-compensated DSA image is predicted by the second neural network that comprises:
a generative model trained to predict, from the inputted temporal sequence of DSA images, a candidate composite motion-compensated DSA image representing the inputted temporal sequence of DSA images (e.g. the generator is used to output or predict an output image from the training dataset images in order to provide image data to the discriminator, which is taught in ¶ [38]-[40] above.); and
wherein the neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image (e.g. the training images are input into the generator and an output DSA image is output to the discriminator, which is taught in ¶ [38]-[40] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the composite motion-compensated DSA image is predicted by the second neural network that comprises: a generative model trained to predict, from the inputted temporal sequence of DSA images, a candidate composite motion-compensated DSA image representing the inputted temporal sequence of DSA images; and wherein the neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 4: (Currently amended) However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 3, wherein the second neural network is trained to predict, from input of the temporal sequence, the composite motion-compensated DSA image, by:
providing a discriminative model, and training the generative model (180) to predict, from the inputted temporal sequence of DSA images, a candidate composite motion- compensated DSA image representing the inputted temporal sequence of DSA images, by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts;
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison;
inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate composite motion-compensated DSA image with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data, and computing a discriminator loss based on the comparison; and
adjusting parameters of the generative model and the discriminative model based on the reconstruction loss, and the discriminator loss, respectively.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the second neural network is trained to predict, from input of the temporal sequence, the composite motion-compensated DSA image, by:
providing a discriminative model, and training the generative model (180) to predict, from the inputted temporal sequence of DSA images, a candidate composite motion- compensated DSA image representing the inputted temporal sequence of DSA images (e.g. a discriminator receives a predicted, or an image without motion artifacts, from the generator. The generator outputs a candidate from the input training images that are comprised of DSA images, which is taught in ¶ [38]-[40] above.), by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37] above.);
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison (e.g. training datasets A and B contain minimal or no motion artifacts that are input into the generator. The DSA images are classified as having artifacts that are input into the generator in order to output a DSA image without motion artifacts. The system calculates a distance loss by comparing the DSA image to a ground truth. This is taught in ¶ [34]-[38], [43] and [45] above.);
inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate composite motion-compensated DSA image with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data, and computing a discriminator loss based on the comparison (e.g. the image output from the generator is input into the discriminator, which is taught in ¶ [37]-[40] above. The discriminator is used to classify whether or not the received image is considered as a real image with no motion artifacts, or a fake image, based on the discriminator training using ground truth outputs, which is taught in ¶ [47].); and
[0047] To evaluate the performance of the proposed methods, the disclosed methods were used to generate outputs using both a quantitative assessment and a qualitative assessment. The quantitative assessment is accomplished by using the testing datasets, which were not included in the training phase, for both datasets A and B. Since these datasets do not contain any motion artifacts, the algorithm outputs can be compared directly to the ground truth DSA images using a variety of full-reference image quality metrics. This quantitative comparison was used to ensure the ability of each algorithm to accurately reproduce the traditional DSA method in settings without motion artifacts. Furthermore, a quantitative comparison was used to analyze the impact of transfer learning from cerebral angiograms, prior to training on the smaller dataset of hepatic angiograms. Results were compared using a two-sided T-test assuming equal population variances.
adjusting parameters of the generative model and the discriminative model based on the reconstruction loss, and the discriminator loss, respectively (e.g. the adversarial loss and the distance loss are used to adjust the parameters of the GAN, which is taught in ¶ [43] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the second neural network is trained to predict, from input of the temporal sequence, the composite motion-compensated DSA image, by: providing a discriminative model, and training the generative model (180) to predict, from the inputted temporal sequence of DSA images, a candidate composite motion- compensated DSA image representing the inputted temporal sequence of DSA images, by: receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts; inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison; inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate composite motion-compensated DSA image with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data, and computing a discriminator loss based on the comparison; and adjusting parameters of the generative model and the discriminative model based on the reconstruction loss, and the discriminator loss, respectively, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 5: (Currently amended) However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 3, comprising at least one of enforcing cycle consistency and/or spatial consistency between the candidate composite motion-compensated DSA image, and a combined image representing the inputted images.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses comprising at least one of enforcing cycle consistency and spatial consistency between the candidate composite motion-compensated DSA image, and a combined image representing the inputted images (e.g. calculating a distance loss is used to that can contribute to the cycle consistency and spatial consistency between an output DSA image and an input ground truth image, which is taught in ¶ [43] above. However, SSIM can be used to determine consistency between a DSA image and the ground truth, which is taught in ¶ [49].).
[0049] When performing the quantitative assessment, the visual quality of the generated images for the cerebral and hepatic angiograms was assessed using full-reference image quality metrics, including the mean-squared error (MSE) calculated from normalized images, the structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR). As shown in Table 2 and 3, generated images were visually similar to the ground truth with adequate MSE, SSIM, and PSNR scores for both the cerebral and hepatic angiograms. More specifically, Table 2 depicts Full-Reference (FR) Image Quality metrics, including MSE, SSIM, and PSNR values, for each algorithm combination for the cerebral testing dataset with no motion artifacts present. Generated images were compared directly to the traditional DSA method for visual similarity in the setting of no motion artifacts. Reported as mean value and 95% confidence interval. Table 3 depicts Full-Reference (FR) Image Quality metrics, including MSE, SSIM, and PSNR values, for each algorithm combination for the hepatic testing dataset with no motion artifacts present. Generated images were compared directly to the traditional DSA method for visual similarity in the setting of no motion artifacts. Reported as mean value and 95% confidence interval.
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of comprising at least one of enforcing cycle consistency and spatial consistency between the candidate composite motion-compensated DSA image, and a combined image representing the inputted images, incorporated in the device of Rauch, in order to use neural networks and training data to calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 13: (Currently amended) Rauch discloses the computer-implemented method according to claim 1, the composite motion-compensated DSA image, the method further comprising
to perform motion compensation on a temporal sequence of digital subtraction angiography (DSA) images generated by subtracting a mask image, from a temporal sequence of contrast- enhanced images (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25] above. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25] above.).
However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 1, wherein the second neural network is trained to predict, from the input of the temporal sequence,
the composite motion-compensated DSA image, the method further comprising training a generative adversarial network (GAN) comprising a generative model and a discriminative model, the method comprising:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts;
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison;
inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting (S230), classifying (S240) the inputted candidate composite motion-compensated DSA image (190) as either having motion artifacts or as not having motion artifacts, by comparing (S250) the inputted candidate composite motion-compensated DSA image (190) with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data (200), and computing a discriminator loss based on the comparison; and
adjusting parameters of the generative model (180) and the discriminative model based on the reconstruction loss and the discriminator loss, respectively.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the second neural network is trained to predict, from the input of the temporal sequence, the composite motion-compensated DSA image (e.g. a neural network is trained in order to determine a DSA image that is motion compensated, which is taught in ¶ [35]-[39] above.), the method further comprising
training a generative adversarial network (GAN) comprising a generative model and a discriminative model, (e.g. generator and discriminative models perform motion correction on a sequence of DSA images, which is taught in ¶ [35]-[39] above.), the method comprising:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37] above.);
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison (e.g. training datasets A and B contain minimal or no motion artifacts that are input into the generator. The DSA images are classified as having artifacts that are input into the generator in order to output a DSA image without motion artifacts. The system calculates a distance loss by comparing the DSA image to a ground truth. This is taught in ¶ [34]-[38], [43] and [45] above.);
inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting (S230), classifying (S240) the inputted candidate composite motion-compensated DSA image (190) as either having motion artifacts or as not having motion artifacts, by comparing (S250) the inputted candidate composite motion-compensated DSA image (190) with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data (200), and computing a discriminator loss based on the comparison (e.g. the image output from the generator is input into the discriminator, which is taught in ¶ [37]-[40] above. The discriminator is used to classify whether or not the received image is considered as a real image with no motion artifacts, or a fake image, based on the discriminator training using ground truth outputs, which is taught in ¶ [47] above.); and
adjusting parameters of the generative model (180) and the discriminative model based on the reconstruction loss and the discriminator loss, respectively (e.g. the adversarial loss and the distance loss are used to adjust the parameters of the GAN, which is taught in ¶ [43] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the second neural network is trained to predict, from the input of the temporal sequence,
the composite motion-compensated DSA image, the method further comprising training a generative adversarial network (GAN) comprising a generative model and a discriminative model, the method comprising:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts;
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a reconstruction loss based on the comparison; inputting the candidate composite motion-compensated DSA image into the discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate composite motion-compensated DSA image with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data, and computing a discriminator loss based on the comparison; and
adjusting parameters of the generative model and the discriminative model based on the reconstruction loss and the discriminator loss, respectively, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 14: (Currently amended) Rauch discloses a non-transitory computer-readable storage medium having stored a computer program product comprising instructions which, when executed by a processor one or more processors (e.g. a processor is used in the subtraction of the mask image, which is taught in ¶ [14] above.), cause the one or more processors processor to:
receive a temporal sequence of DSA images of a vasculature generated by subtracting a mask image from a temporal sequence of contrast-enhanced images (e.g. images of DSA image sequence is generated based on subtracting a mask image from a shifted image to produce a subtracted image enhancing vessel structure, which is taught in ¶ [14], [19] and [20] above.);
temporal sequence of motion-compensated DSA images (150) arising from corresponding motion of the vasculature between successive contrast- enhanced images in the temporal sequence (e.g. the temporal sequence of motion-compensated DSA images come from the vasculature between successive contrast-enhanced images in the temporal sequence, which is taught in ¶ [21]-[25] above.),
predict, based on motion artifacts in the DSA images, a composite motion-compensated DSA image representing the temporal sequence of DSA image and including (i) compensation for motion of the vasculature between successive contrast-enhanced images in the temporal sequence and (ii) compensation for motion of the vasculature between acquisition of the contrast-enhanced images in the temporal sequence and acquisition of the mask image (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25] above. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25] above.); and
output the predicted composite motion-compensated DSA image (e.g. the determined composite motion-compensated DSA image can be output to a display, which is taught in ¶ [29] and [30] above.).
However, Rauch fails to specifically teach the features of input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image;
input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image (e.g. trained dataset C images are input into the generator in order to output a corresponding motion compensated DSA image that is corrected for the motion between an initial mask and overall input image. This is taught in ¶ [37]-[41].and [51]-[54] above. When using the PatchGan, the patches of a part of an image is used for evaluation. The generator is used to compensate for the motion artifacts of the patch DSA images within the dataset C. The patches of the image reflect a temporal sequence of motion compensated DSA images that are output to the discriminative model. The DSA image is in between the post contrast image and the subtraction of the mask image, which is taught in ¶ [30] above.);
input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images (e.g. the output of the generator comprises motion-compensated patches that are input into the discriminative model, which is the second neural network. The discriminative model judges whether each patch is real or fake and averages this result. Once the patches are composed to make up an overall image, which is considered as a composite image, the system makes an ultimate real or fake decision on the overall image. This is taught in ¶ [37]-[41] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image; input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 15: (Currently amended) Rauch discloses a system for performing motion compensation on a temporal sequence of digital subtraction angiography (DSA), the system comprising:
a processor communicatively coupled to a memory (e.g. a processor is contains a memory that is used to perform the invention tasks, which is taught in ¶ [29].), the processor configured to:
receive a temporal sequence of DSA images of a vasculature generated by subtracting a mask image from a temporal sequence of contrast-enhanced images (e.g. images of DSA image sequence is generated based on subtracting a mask image from a shifted image to produce a subtracted image enhancing vessel structure, which is taught in ¶ [14], [19] and [20] above.);
temporal sequence of motion-compensated DSA images (150) arising from corresponding motion of the vasculature between successive contrast- enhanced images in the temporal sequence (e.g. the temporal sequence of motion-compensated DSA images come from the vasculature between successive contrast-enhanced images in the temporal sequence, which is taught in ¶ [21]-[25] above.),
predict, based on motion artifacts in the DSA images, a composite motion- compensated DSA image representing the temporal sequence of DSA image and including (i) compensation for motion of the vasculature between successive contrast-enhanced images in the temporal sequence and (ii) compensation for motion of the vasculature between acquisition of the contrast-enhanced images in the temporal sequence and acquisition of the mask image (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25] above. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25] above.); and
output the predicted composite motion-compensated DSA image (e.g. the determined composite motion-compensated DSA image can be output to a display, which is taught in ¶ [29] and [30] above.).
However, Rauch fails to specifically teach the features of input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image;
input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image (e.g. trained dataset C images are input into the generator in order to output a corresponding motion compensated DSA image that is corrected for the motion between an initial mask and overall input image. This is taught in ¶ [37]-[41].and [51]-[54] above. When using the PatchGan, the patches of a part of an image is used for evaluation. The generator is used to compensate for the motion artifacts of the patch DSA images within the dataset C. The patches of the image reflect a temporal sequence of motion compensated DSA images that are output to the discriminative model. The DSA image is in between the post contrast image and the subtraction of the mask image, which is taught in ¶ [30] above.);
input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images (e.g. the output of the generator comprises motion-compensated patches that are input into the discriminative model, which is the second neural network. The discriminative model judges whether each patch is real or fake and averages this result. Once the patches are composed to make up an overall image, which is considered as a composite image, the system makes an ultimate real or fake decision on the overall image. This is taught in ¶ [37]-[41] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of input the temporal sequence of DSA images into a first neural network, wherein the first neural network is configured to output a corresponding temporal sequence of motion-compensated DSA images, wherein the motion-compensated DSA images include compensation for motion of the vasculature between the acquisition of each contrast-enhanced image in the temporal sequence and the acquisition of the mask image; input the corresponding temporal sequence of motion-compensated DSA images into a second neural network, wherein the second neural network predicts a composite motion-compensated DSA image that represents the predicted temporal sequence of motion-compensated DSA images and includes compensation for motion of the vasculature in the predicted temporal sequence of motion-compensated DSA images, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 16: (New) However, Rauch fails to specifically teach the features of the non-transitory computer-readable storage medium according to claim 14, wherein the instructions, when executed by the processor, further cause the processor to: apply a machine-learning model to predict the composite motion-compensated DSA image, the machine-learning model trained based a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the instructions, when executed by the processor, further cause the processor to: apply a machine-learning model to predict the composite motion-compensated DSA image (e.g. a neural network is trained with the input of DSA images, in order to predict or output a DSA image without motion artifacts, which is taught in ¶ [38]-[40] above.), by:
the machine-learning model trained based a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37].).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the instructions, when executed by the processor, further cause the processor to: apply a machine-learning model to predict the composite motion-compensated DSA image, the machine-learning model trained based a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 17: (New) However, Rauch fails to specifically teach the features of the system according to claim 15, wherein the processor is further configured to: apply a machine-learning model to predict the composite motion-compensated DSA image, the machine-learning model trained based on a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the processor is further configured to: apply a machine-learning model to predict the composite motion-compensated DSA image (e.g. a neural network is trained with the input of DSA images, in order to predict or output a DSA image without motion artifacts, which is taught in ¶ [38]-[40] above.), by:
the machine-learning model trained based on a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37].).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the processor is further configured to: apply a machine-learning model to predict the composite motion-compensated DSA image, the machine-learning model trained based on a plurality of DSA images of the vasculature classified as not having motion artifacts and a plurality of DSA images of the vasculature classified as having motion artifacts, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Claim(s) 7-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rauch, as modified by the features of Crabb, as applied to claims 1, 14 and 15 above and further in view of Sun (US Pub 2021/0133984).
Re claim 7: However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 1, wherein the first neural network comprises: a first generative model trained to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image;
wherein the second neural network comprises: a second generative model configured to receive the candidate DSA images predicted by the first generative model, and to predict, from the received candidate DSA images, a candidate composite motion- compensated DSA image representing the received candidate DSA images, and including compensation for motion of the vasculature between successive contrast-enhanced images in the received candidate DSA images; and
wherein the second neural network (120) is configured to output the candidate composite motion-compensated DSA image (190) to provide the composite motion-compensated DSA image (130).
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses wherein the first neural network comprises: a first generative model trained to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image (e.g. trained dataset images are input into the generator in order for training to predict a corresponding motion compensated DSA image that is corrected for the motion between an initial mask and overall input image. This is taught in ¶ [37]-[41] above.);
wherein the second neural network comprises: a second generative model configured to receive the candidate DSA images predicted by the first generative model, and to predict, from the received candidate DSA images, a candidate composite motion- compensated DSA image representing the received candidate DSA images, and including compensation for motion of the vasculature between successive contrast-enhanced images in the received candidate DSA images (e.g. the discriminative model is considered as the second neural network that receives patches of motion compensated DSA images from the generator. The discriminative model evaluates whether the patches are real or fake while gathering the patch images into an overall image. Once the overall image is formed and after averaging the results of the evaluation of the patches, the system gives an overall final real or fake determination on the overall image, which is taught in ¶ [37]-[41] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the first neural network comprises: a first generative model trained to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image; wherein the second neural network comprises: a second generative model configured to receive the candidate DSA images predicted by the first generative model, and to predict, from the received candidate DSA images, a candidate composite motion- compensated DSA image representing the received candidate DSA images, and including compensation for motion of the vasculature between successive contrast-enhanced images in the received candidate DSA images, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
However, the combination above fails to specifically teach the features of wherein the second neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses use a GAN for image enhancement (same field of endeavor or reasonably pertinent to the problem).
Sun discloses wherein the second neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image (e.g. the second generator outputs the candidate image to a discriminator, which is taught in ¶ [122]-[124].).
[0121] In some embodiments, the preliminary model may be a preliminary model 1200 as illustrated in FIG. 12. The preliminary model 1200 may be trained using the same or similar training sample(s) of the preliminary model 1100 to generate a second trained model. The preliminary model 1200 may include a forward pipeline (the left portion illustrated in FIG. 12) and a backward pipeline (the right portion illustrated in FIG. 12). Each of the forward and backward pipelines may include the same or similar configuration as the preliminary model 1100 as described in connection with FIG. 11. For example, as shown in FIG. 12, the forward pipeline may include a generator 1102A, a transformation layer 1104A, and a discriminator 1105A. The backward pipeline may include a generator 1102B, a transformation layer 1104B, and a discriminator 1105B.
[0122] For the training sample 1101, the image pair (A, B) and the image pair (B, A) may be inputted into the forward pipeline and the backward pipeline, respectively. The forward pipeline may be used to generate an image B′ (i.e., a predicted image B) by warping the image A according to the image B. For example, the generator 1102A may predict a motion field 1103A from the image A to the image B of the training sample 1101. The transformation layer 1104A may generate the image B′ by warping the image A according to the motion field 1103A. The discriminator 1105A may be configured to generate a discrimination result between the images B and B′. The backward pipeline may be used to generate an image A′ (i.e., a predicted image A) by warping the image B according to the image A. For example, the generator 1102B may predict a motion field 1103B from the image B to the image A of the training sample 1101. The transformation layer 1104B may generate the image A′ by warping the image B according to the motion field 1103B. The discriminator 1105B may be configured to generate a discrimination result between the images A and A′.
[0123] In some embodiments, the preliminary model 1200 may be trained to minimize a loss function of the preliminary model 1200. The loss function of the preliminary model 1200 may include a first component associated with the forward pipeline and/or a second component associated with the backward pipeline. Each of the first component and the second component may be similar to the loss function of the preliminary model 1100 as described in connection with FIG. 11. Taking the forward pipeline as an instance, the corresponding first component may relate to a first difference and a discrimination result between images B and B′ of each training sample, and optionally a consistency motion loss.
[0124] In some embodiments, for the training sample 1101, the processing device 140B may further replace the image A in the image pair (A, B) with the image A′ generated by the back pipeline to generate an image pair (A′, B). The image pair (A′, B) may be inputted into the forward pipeline to generate an image B″ (or also referred be to as a third image) by warping the image A′ according to the image B of the training sample 1101. In other words, the image A′ may be generated by performing a backward transformation on the image B of the training sample 1101, and the image B″ may be generated by performing a forward transformation on the image A′ of the training sample 1101. Theoretically, if the preliminary model 1200 is accurate enough (e.g., having an accuracy higher than a threshold), the image B″ may be substantially the same as the image B of the training sample 1101. A third difference between the images B and B″ may be determined and taken into consideration in the training of the preliminary model 1200. For example, the loss function of the preliminary model 1200 may be determined based on the first component relating to the forward pipeline, the second component relating to the backward pipeline, the third difference of each training sample, or any combination thereof.
Therefore, in view of Sun, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the second neural network is configured to output the candidate composite motion-compensated DSA image to provide the composite motion-compensated DSA image, incorporated in the device of Rauch, as modified by Crabb, in order to use multiple neural networks to predict motion compensated image data, which can improve accuracy and reliability of a second trained model when using multiple iterations (as stated in Sun ¶ [125]).
Re claim 8: (Currently amended) Rauch discloses the computer-implemented method according to claim 1,
a candidate DSA image (170) that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image (e.g. the system can determine a composite motion-compensated DSA image that represents a sequence of the DSA image by compensating for motion between the subtracted mask image and contrast enhanced image. In addition, the compensation between contrast enhanced images can be included when determining composite motion-compensated images, which is taught in ¶ [12] and [21]-[25] above. The production of the DSA images is performed by using or reducing the motion, or motion related artifacts, that are associated with a foreground and background, which is taught in ¶ [12] and [25] above.).
However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 6, wherein the second neural network (120) is trained to predict, from the inputted temporal sequence (110), the composite motion-compensated DSA image (130), by:
providing a first discriminative model (240), and training the first generative model (160) to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image (170) that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image, by:
receiving DSA training image data (200) including a plurality of DSA images of the vasculature classified as having motion artifacts (220), and a plurality of DSA images of the vasculature classified as not having motion artifacts (210);
inputting, from the received DSA training image data (200), the DSA images of the vasculature classified as having motion artifacts (220) into the first generative model (160), and in response to the inputting, generating for each inputted image, a candidate DSA image (170) that includes compensation for motion of the vasculature between the acquisition of the corresponding contrast-enhanced image and the acquisition of the mask image, by comparing each generated candidate DSA image (170) with the corresponding inputted DSA image of the vasculature from the received DSA training image data (200), and computing a first reconstruction loss based on the comparison;
inputting the candidate DSA image (170) into the first discriminative model (240), and in response to the inputting, classifying the inputted candidate DSA image (170) as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate DSA image (170) with one or more DSA images of the vasculature classified as not having motion artifacts (210) from the DSA training image data (200), and computing a first discriminator loss based on the comparison;
adjusting parameters of the first generative model (160) and the first discriminative model (240) based on the first reconstruction loss and the first discriminator loss, respectively;
providing a second discriminative model (230), and training the second generative model (180) to predict a candidate composite motion-compensated DSA image (190), by:
inputting the temporal sequence of candidate DSA images (170) generated by the first generative model (160) into the second generative model (180), and in response to the inputting, generating a candidate composite motion-compensated DSA image (190) by comparing the generated composite motion-compensated DSA image (190) with a combined image representing the inputted images, and computing a second reconstruction loss based on the comparison;
inputting the candidate composite motion-compensated DSA image into the second discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by using the second discriminative model to compare the inputted candidate composite motion-compensated DSA image with one or more DSA images classified as not having motion artifacts from the DSA training image data, and computing a second discriminator loss based on the comparison; and
adjusting parameters of the second generative model (180) and the second discriminative model (230) based on the second reconstruction loss, and the second discriminator loss, respectively.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses providing a first discriminative model, and training the first generative model to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image (e.g. a discriminator receives a predicted, or an image without motion artifacts, from the generator. The generator outputs a candidate from the input training images that are comprised of DSA images with motion artifacts that are motion corrected as an output, which is taught in ¶ [38]-[40] above. The correction occurs to the contrast enhanced image to the DSA image.),
by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts (e.g. the datasets A and B contain DSA images with minimal or no motion artifacts that are used to train the GAN, which is taught in ¶ [34]-[37] above.);
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the first generative model, and in response to the inputting, generating for each inputted image, a candidate DSA image that includes compensation for motion of the vasculature between the acquisition of the corresponding contrast-enhanced image and the acquisition of the mask image, by comparing each generated candidate DSA image with the corresponding inputted DSA image of the vasculature from the received DSA training image data, and computing a first reconstruction loss based on the comparison (e.g. training datasets A and B contain minimal or no motion artifacts that are input into the generator. The DSA images are classified as having artifacts that are input into the generator in order to output a DSA image without motion artifacts. The system calculates a distance loss by comparing the DSA image to a ground truth. This is taught in ¶ [34]-[38], [43] and [45] above.);
inputting the candidate DSA image into the first discriminative model (240), and in response to the inputting, classifying the inputted candidate DSA image (170) as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate DSA image (170) with one or more DSA images of the vasculature classified as not having motion artifacts (210) from the DSA training image data (200), and computing a first discriminator loss based on the comparison (e.g. the image output from the generator is input into the discriminator, which is taught in ¶ [37]-[40] above. The discriminator is used to classify whether or not the received image is considered as a real image with no motion artifacts, or a fake image, based on the discriminator training using ground truth outputs, which is taught in ¶ [47] above.);
adjusting parameters of the first generative model (160) and the first discriminative model (240) based on the first reconstruction loss and the first discriminator loss, respectively (e.g. the adversarial loss and the distance loss are used to adjust the parameters of the GAN, which is taught in ¶ [43] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the second neural network is trained to predict, from the inputted temporal sequence, the composite motion-compensated DSA image, by:
providing a first discriminative model, and training the first generative model to predict, for each inputted DSA image in the temporal sequence, a candidate DSA image that includes compensation for the motion of the vasculature between the acquisition of the corresponding contrast-enhanced image in the temporal sequence and the acquisition of the mask image, by:
receiving DSA training image data including a plurality of DSA images of the vasculature classified as having motion artifacts, and a plurality of DSA images of the vasculature classified as not having motion artifacts;
inputting, from the received DSA training image data, the DSA images of the vasculature classified as having motion artifacts into the first generative model, and in response to the inputting, generating for each inputted image, a candidate DSA image that includes compensation for motion of the vasculature between the acquisition of the corresponding contrast-enhanced image and the acquisition of the mask image, by comparing each generated candidate DSA image with the corresponding inputted DSA image of the vasculature from the received DSA training image data, and computing a first reconstruction loss based on the comparison;
inputting the candidate DSA image into the first discriminative model, and in response to the inputting, classifying the inputted candidate DSA image as either having motion artifacts or as not having motion artifacts, by comparing the inputted candidate DSA image with one or more DSA images of the vasculature classified as not having motion artifacts from the DSA training image data (200), and computing a first discriminator loss based on the comparison; adjusting parameters of the first generative model and the first discriminative model based on the first reconstruction loss and the first discriminator loss, respectively, incorporated in the device of Rauch, in order to use neural networks and training data calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
However, the combination above fails to specifically teach the features of providing a second discriminative model, and training the second generative model to predict a candidate composite motion-compensated DSA image, by:
inputting the temporal sequence of candidate DSA images generated by the first generative model into the second generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a second reconstruction loss based on the comparison;
inputting the candidate composite motion-compensated DSA image into the second discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by using the second discriminative model to compare the inputted candidate composite motion-compensated DSA image with one or more DSA images classified as not having motion artifacts from the DSA training image data, and computing a second discriminator loss based on the comparison; and
adjusting parameters of the second generative model (180) and the second discriminative model (230) based on the second reconstruction loss, and the second discriminator loss, respectively.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses use a GAN for image enhancement (same field of endeavor or reasonably pertinent to the problem).
Sun discloses the providing a second discriminative model, and training the second generative model to predict a candidate composite motion-compensated DSA image (e.g. a second generator and discriminator are trained in order to predict a motion compensated image, which is taught in ¶ [121] above.), by:
inputting the temporal sequence of candidate DSA images generated by the first generative model into the second generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a second reconstruction loss based on the comparison (e.g. the previously applied references disclose using DSA images and inputting these images into a GAN. The Su reference can be used with DSA-MRI system, which is taught in ¶ [45]. Predicted images from a first generator can be input into a second generator in order to generate a motion compensated image for the generation of a loss based on comparing the motion compensated image with an image determined from a sequence of images. This is taught in ¶ [121]-[124] above.);
[0045] Provided herein are systems and methods for non-invasive biomedical imaging, such as for disease diagnostic or research purposes. In some embodiments, the systems may include a single modality imaging system and/or a multi-modality imaging system. The single modality imaging system may include, for example, an ultrasound imaging system, an X-ray imaging system, an computed tomography (CT) system, a magnetic resonance imaging (MRI) system, an ultrasonography system, a positron emission tomography (PET) system, an optical coherence tomography (OCT) imaging system, an ultrasound (US) imaging system, an intravascular ultrasound (IVUS) imaging system, a near infrared spectroscopy (NIRS) imaging system, a far infrared (FIR) imaging system, or the like, or any combination thereof. The multi-modality imaging system may include, for example, an X-ray imaging-magnetic resonance imaging (X-ray-MRI) system, a positron emission tomography-X-ray imaging (PET-X-ray) system, a single photon emission computed tomography-magnetic resonance imaging (SPECT-MRI) system, a positron emission tomography-computed tomography (PET-CT) system, a C-arm system, a digital subtraction angiography-magnetic resonance imaging (DSA-MRI) system, etc. It should be noted that the imaging system described below is merely provided for illustration purposes, and not intended to limit the scope of the present disclosure.
inputting the candidate composite motion-compensated DSA image into the second discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by using the second discriminative model to compare the inputted candidate composite motion-compensated DSA image with one or more DSA images classified as not having motion artifacts from the DSA training image data, and computing a second discriminator loss based on the comparison (e.g. Similar to the Crabb reference, the Sun reference can employ a patch GAN discriminator that can determine if the image data received by the second discriminator has a probability of being the real image without artifacts. The second discriminator loss is calculated based on a comparison between the motion compensated image predicted and an image that is determined from a sequence of images, which is taught in ¶ [121]-[124] above and [120].); and
[0120] In some embodiments, the preliminary model 1100 may further include a discriminator 1105 as shown in FIG. 11. Such preliminary model 1100 including the discriminator 1105 may also be referred to as a generative adversarial network (GAN) model. For the training sample 1101, the discriminator 1105 may be configured to receive the images B and B′ of the training sample 1101, and discern which one is a real image to generate a discrimination result between the images B and B′. For example, the discriminator result may include a determination as to whether the image B is a real image, a probability that the image B is a real image, a determination as to whether the image B′ is a real image, a probability that the image B′ is a real image, or the like, or any combination thereof. In some embodiments, the discriminator 1105 may be any neural network component that can realize its function. Merely by way of example, the discriminator 1105 may be an image classifier, a patch GAN discriminator, etc. In some embodiments, the loss function of a preliminary model 1100 that includes the discriminator 1105 may be determined based on the first difference, the discrimination result, and optionally the second difference of each training sample. In some embodiments, the loss function of a preliminary model 1100 that includes the discriminator 1105 may be a GAN loss.
adjusting parameters of the second generative model and the second discriminative model based on the second reconstruction loss, and the second discriminator loss, respectively (e.g. the second component loss is used to adjust the second generator and discriminator, which is taught in ¶ [122]-[124] above.).
Therefore, in view of Sun, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of providing a second discriminative model, and training the second generative model to predict a candidate composite motion-compensated DSA image, by: inputting the temporal sequence of candidate DSA images generated by the first generative model into the second generative model, and in response to the inputting, generating a candidate composite motion-compensated DSA image by comparing the generated composite motion-compensated DSA image with a combined image representing the inputted images, and computing a second reconstruction loss based on the comparison; inputting the candidate composite motion-compensated DSA image into the second discriminative model, and in response to the inputting, classifying the inputted candidate composite motion-compensated DSA image as either having motion artifacts or as not having motion artifacts, by using the second discriminative model to compare the inputted candidate composite motion-compensated DSA image with one or more DSA images classified as not having motion artifacts from the DSA training image data, and computing a second discriminator loss based on the comparison; and adjusting parameters of the second generative model (180) and the second discriminative model (230) based on the second reconstruction loss, and the second discriminator loss, respectively, incorporated in the device of Rauch, as modified by Crabb, in order to use multiple neural networks to predict motion compensated image data, which can improve accuracy and reliability of a second trained model when using multiple iterations (as stated in Sun ¶ [125]).
Re claim 9. (Currently amended) However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 8, comprising enforcing cycle consistency and/or spatial consistency between the candidate DSA image, and the corresponding inputted image from the received DSA training image data.
However, this is well known in the art as evidenced by Crabb. Similar to the primary reference, Crabb discloses correcting digital subtraction angiography images (same field of endeavor or reasonably pertinent to the problem).
Crabb discloses comprising enforcing cycle consistency and/or spatial consistency between the candidate DSA image, and the corresponding inputted image from the received DSA training image data (e.g. calculating a distance loss is used to that can contribute to the cycle consistency and spatial consistency between an output DSA image and an input ground truth image, which is taught in ¶ [43] above. However, SSIM can be used to determine consistency between a DSA image and the ground truth, which is taught in ¶ [49] above.).
Therefore, in view of Crabb, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of comprising enforcing cycle consistency and/or spatial consistency between the candidate DSA image, and the corresponding inputted image from the received DSA training image data, incorporated in the device of Rauch, in order to use neural networks and training data to calculate loss function variables to contribute to the change of a GAN, which allows for production of enhanced medical images (as stated in Crabb ¶ [28]).
Re claim 10: (Currently amended) However, the combination above fails to specifically teach the features of the computer-implemented method according to claim 8, wherein at least some of the parameters of the first discriminative model of the first neural network are common to the first discriminative model of the first neural network and the second discriminative model of the second neural network.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses use a GAN for image enhancement (same field of endeavor or reasonably pertinent to the problem).
Sun discloses wherein at least some of the parameters of the first discriminative model of the first neural network are common to the first discriminative model of the first neural network and the second discriminative model of the second neural network (e.g. the loss function of the overall model can include the first component associated with the forward pipeline and the second component associated with the backward pipeline, which is taught in ¶ [122]-[125] above.).
Therefore, in view of Sun, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein at least some of the parameters of the first discriminative model of the first neural network are common to the first discriminative model of the first neural network and the second discriminative model of the second neural network, incorporated in the device of Rauch, as modified by Crabb, in order to use multiple neural networks to predict motion compensated image data, which can improve accuracy and reliability of a second trained model when using multiple iterations (as stated in Sun ¶ [125]).
Re claim 11: (Currently amended) However, Rauch fails to specifically teach the features of the computer-implemented method according to claim 8, wherein the adjusting parameters of the first generative model and the first discriminative model of the first neural network is based further on the classification provided by the second discriminative model of the second neural network.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses use a GAN for image enhancement (same field of endeavor or reasonably pertinent to the problem).
Sun discloses wherein the adjusting parameters of the first generative model and the first discriminative model of the first neural network is based further on the classification provided by the second discriminative model of the second neural network (e.g. the loss function is adjusted based on the loss associated with the first component, the second component or a combination in order to change the parameters of the first model. This is taught in ¶ [121]-[125] above.).
Therefore, in view of Sun, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the adjusting parameters of the first generative model and the first discriminative model of the first neural network is based further on the classification provided by the second discriminative model of the second neural network, incorporated in the device of Rauch, as modified by Crabb, in order to use multiple neural networks to predict motion compensated image data, which can improve accuracy and reliability of a second trained model when using multiple iterations (as stated in Sun ¶ [125]).
Re claim 12: (Currently amended) However, the combination above fails to specifically teach the features of the computer-implemented method according to claim 8, comprising receiving user input indicative of a region of interest in the received DSA training image data (200); and applying a weighting to the reconstruction loss and/or to the discriminator loss such that a higher weighting is applied within the region of interest than outside the region of interest.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses use a GAN for image enhancement (same field of endeavor or reasonably pertinent to the problem).
Sun discloses comprising receiving user input indicative of a region of interest in the received DSA training image data (200) (e.g. a user can indicate a region of interest that is selected for segmentation, which is taught in ¶ [73[, [88] and [89].); and
[0073] The identification module 402 may be configured to identify one or more feature points relating to the ROI from the reference image. A feature point may refer to a representative point of the ROI which can be used to measure the physiological motion of the ROI. In some embodiments, the feature point(s) may be identified from the reference image according to a user input, or by the processing device 140A automatically or semi-automatically. More descriptions regarding the identification of the feature point(s) may be found elsewhere in the present disclosure. See, e.g., operation 502 in FIG. 5 and relevant descriptions thereof.
[0088] In some embodiments, the feature point(s) may be identified from the reference image according to a user input. For example, via a user interface implemented on, e.g., a terminal 130 or a mobile device 300, a user may manually mark one or more feature points in the reference image. Alternatively, the feature point(s) may be identified from the reference image automatically by the processing device 140A. Alternatively, the feature point(s) may be identified from the reference image by the processing device 140A semi-automatically. For example, the feature point identification may be performed by the processing device 140A based on an image analysis algorithm in combination with user intervention. Exemplary user interventions may include providing a parameter relating to the image analysis algorithm, providing a position parameter relating to a feature point, making an adjustment to or confirming a preliminary feature point identified by the processing device 140A, providing instructions to cause the processing device 140A to repeat or redo the feature point identification, etc.
[0089] For illustration purposes, an exemplary process for identifying an inner point and a corresponding outer point of the heart from a reference mage is provided hereinafter. In some embodiments, the processing device 140A may segment an endocardium and an epicardium of the heart from the reference image. For example, a user may manually annotate the endocardium and the epicardium from the reference image via a user interface, and the processing device 140A may segment the endocardium and the epicardium according to the user annotation. As another example, the endocardium and the epicardium of the myocardium may be segmented by the processing device 140A automatically according to an image analysis algorithm (e.g., an image segmentation algorithm). Alternatively, the endocardium and the epicardium may be segmented by the processing device 140A semi-automatically based on an image analysis algorithm in combination with information provided by a user. Exemplary information provided by the user may include a parameter relating to the image analysis algorithm, a position parameter relating to the endocardium and the epicardium, an adjustment to or confirmation of a preliminary endocardium and/or a preliminary epicardium generated by the processing device 140A, etc.
applying a weighting to the reconstruction loss and/or to the discriminator loss such that a higher weighting is applied within the region of interest than outside the region of interest (e.g. a loss is calculated and minimized based on the region of interest selected by the user being compared to another motion image. The weighting is considered as parameters that are updated to minimize loss, which is taught in ¶ [97], [118] and [133]. In addition, the user can adjust parameters associated with an algorithm used for image analysis or position parameter, which is taught in ¶ [88] and [89] above.).
[0097] The computing device may further generate the motion prediction model by training a first preliminary model using the annotated training sample(s) according to a supervised learning technique. Merely by way of example, the first and second annotated images of each annotated training sample may be inputted into the first preliminary model, which may output a predicted motion field from the first annotated image to the second annotated image. The computing device may determine a value of a first loss function based on the predicted motion field and the known sample motion field of each annotated training sample. For example, the first loss function may measure the difference(s) between the predicted motion field and the sample motion field of the annotated training sample(s). Alternatively, for each training sample, the computing device may determine a predicted motion field and an actual motion field of the first sample feature point(s) from the first motion phase to the second motion phase based on the predicted motion field and the sample motion field of the entire first annotated image, respectively. The first loss function may measure a difference between the predicted and actual motion fields of the first sample feature point(s) of each annotated training sample. The first preliminary model may be iteratively trained to minimize the first loss function. The trained model of the first preliminary model may be designated as the motion prediction model.
[0118] In some embodiments, the loss function of the preliminary model 1100 may relate to a first difference between the image B and the image B′ of each training sample. For example, the training sample(s) may include a plurality of training samples. The loss function may be used to measure an overall level (e.g., an average value) of the first differences of the training samples. The loss function may be minimized in the model training so that the first differences between the images B and B′ of the training samples may be minimized locally or globally. As used herein, a difference between two images may be measured by any metrics for measuring a similarity degree or a difference between the two images. Merely by way of example, the difference between two images may be determined based on an image similarity algorithm, including a peak signal to noise ratio (PSNR) algorithm, a structural similarity (SSIM) algorithm, a perceptual hash algorithm, a cosine similarity algorithm, a histogram-based algorithm, a Euclidean distance algorithm, or the like, or any combination thereof.
[0133] As described in connection with FIG. 6, in some embodiments, the motion prediction model may be generated by training a preliminary model using one or more training samples. Each training sample may include a pair of images A and B corresponding to different motion phases of a sample ROI. In some embodiments, the preliminary model may include one or more model parameters having one or more initial values before model training. In the training of the preliminary model, the value(s) of the model parameter(s) of the preliminary model may be updated such that the loss function of the preliminary model may be minimized. In some embodiments, the training of the preliminary model may include one or more iterations. For illustration purposes, a current iteration of the iteration(s) is described in the following description. The current iteration may include one or more operations of process 700 illustrated in FIG. 7.
Therefore, in view of Sun, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of comprising receiving user input indicative of a region of interest in the received DSA training image data (200); and applying a weighting to the reconstruction loss and/or to the discriminator loss such that a higher weighting is applied within the region of interest than outside the region of interest, incorporated in the device of Rauch, as modified by Crabb, in order to use multiple neural networks to predict motion compensated image data, which can improve accuracy and reliability of a second trained model when using multiple iterations (as stated in Sun ¶ [125]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Xu discloses a first and second generator in a GAN that outputs a result into a discriminator.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682
Prosecution Insights