DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 2, filed 3/31/2026, with respect to the 112 (b) rejection have been fully considered and are persuasive. The 112 (b) rejection of the claims has been withdrawn.
Applicant's arguments filed 3/31/2026 have been fully considered but they are not persuasive. The arguments state that the invention does not disclose “review of a single medical scan that contains multiple images from a single scan”. The Examiner disagrees with this assertion based on the Lyman reference. It is accurate that both Conjeti and Lyman perform the feature of having a list of different patients and different scans that can be ordered and reviewed, which is why both are combined. However, Lyman discloses an additional step of the same doctor using the system to review slices of a single scan, which is taught in ¶ [54]. The invention of Lyman discloses allowing for the review of ordered slices of a scan that are ordered based on a probability determined by a model in order to evaluate an abnormality within the center slices and other selected slices associated with the high probability slice, which is taught in ¶ [106], [129], [130], [259]-[261] and [265]-[268]. The medical image can be a plurality of slices of the scan in order to be evaluated in an ordered manner determined by probability of the slice associated with an abnormality, and a workflow of the viewed slices is seen based on the medical professional viewing thumbnails or going through the arranged order of slices. Therefore, based on the above, the features of the independent claims are performed by the combined references.
Thus, based on the above, the rejection of the claims is maintained in view of the previously applied references.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 15-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Conjeti (EP 4020492 A1 (Pub Date: 6/29/2022)) in view of Lyman (US Pub 2020/0111561).
Re claim 1: A computer implemented method for analysing medical scan images from a single patient comprising the steps of:
receiving an input of a plurality of medical scan images for the single patient (e.g. the invention discloses acquiring multiple X-ray images of a patient, which is taught in ¶ [39].):
[0039] For example, a preferred method comprises the following steps:
acquire a first image, e.g. an X-ray image
optionally: send the first image to a determination system, especially via DICOM communication,
optionally: pre-processing of the first image, especially normalization or de-noising,
segmentation of the first image (e.g. of the lung area), especially using a (deep) machine learning algorithm,
examining the first image for abnormalities (e.g. a pneumothorax), especially using a (deep) machine learning algorithm, and in the case an abnormality is found
a) read patient identifier, e.g. from a DICOM header of the first image,
b) query an image archive (such as e.g. PACS) whether a prior image of the same patient in a given time frame (e.g. same day, same week, etc.) is available,
c) if yes, fetch this prior image as second image,
optionally: send the second image to the determination system, especially via DICOM communication,
optionally: pre-processing of the second image, especially normalization or de-noising,
segmentation of the second image (the same area as segmented in the first image), especially using a (deep) machine learning algorithm, preferably using the same algorithm as for the first image,
examining the second image for abnormalities (the same as for the first image), especially using a (deep) machine learning algorithm,
optional: registering both images,
compare abnormalities of the first image with corresponding abnormalities or sections (if there is no abnormality) of the second image, e.g. a pneumothorax relative to the ratio of the lung volume,
determining a value for the change of the abnormality between the images (assessment),
in the case the change exceeds a predefined threshold, e.g. if the percentage affected area on the first image is bigger (by a given offset) than in the second image, flag the images of the patient and prioritize the patient in a worklist.
[0050] Preferred technical components for the invention are: an image viewer such as PACS, a worklist (e.g. PACS- or RIS-driven), AI software (software for a machine learning algorithm) to perform computation, computational hardware for AI software (either cloud-based or on-premise).
[0059] The imaging system (here the CT system 1) records images I1, 12 that are processed by the determination system 6 according to the invention. The determination system 6 comprises a data interface 8, an examination unit 9 and a determination unit 10 to perform the steps of the method according to the invention.
[0060] The data interface 8 is designed for receiving images I1, I2 (see e.g. figure 2 or 3) both showing the same predefined anatomical region TH, e.g. the thorax TH (see e.g. figure 3) of the patient P. The examination unit 9 is designed for examining the received images I1, I2 for abnormalities A of a predetermined type. The determination unit 10 is designed for different purposes of the method.
providing an input of one or more characterised features from the medical scan images, in addition to the plurality of medical scan images to an optimisation model (e.g. a machine learning algorithm is used to evaluate first and second images to determine the abnormalities present within the images. The abnormalities found within the deep machine learning model can be considered as input into the device, which is taught in ¶ [39] above. In addition, the abnormalities are passed to the determination unit with the images in order to determine changes of the abnormality. The determination unit uses a decision algorithm to perform this feature, which is taught in ¶ [60]-[62] and [69].),
[0060] The data interface 8 is designed for receiving images I1, I2 (see e.g. figure 2 or 3) both showing the same predefined anatomical region TH, e.g. the thorax TH (see e.g. figure 3) of the patient P. The examination unit 9 is designed for examining the received images I1, I2 for abnormalities A of a predetermined type. The determination unit 10 is designed for different purposes of the method.
[0061] First, the determination unit 10 is designed for determining changes C of an abnormality A between the first image I1 and the second image 12. Then, the determination unit 10 is designed for assessing the determined changes C and furthermore, the determination unit 10 is designed for determining whether the assessed changes C exceed a predefined threshold T by using a decision algorithm D.
[0062] The output unit 11 is designed for triggering an alert notification N and/or to change a worklist prioritization concerning the respective patient P in the case the assessed changes C exceed the predefined threshold T.
[0069] In step IV, it is determined whether the assessed changes C exceed a predefined threshold T. This is done by using a decision algorithm D, especially a trained machine learning algorithm.
where the optimisation model outputs the characterised features and medical scan images (e.g. the invention is directed towards organizing the reading worklist of the chest x-rays of a patient over time and changing this patient’s images in the workflow order for review based on the detected abnormalities, which is taught in ¶ [39] above and [09].); and
[0009] A method according to the invention for automatically determining (alertable) changes of a condition of a patient, especially generating or organizing a medical reading worklist, preferably for longitudinal triaging of chest x-rays, comprises the following steps:
Receiving at least a first image and a second image, both showing the same predefined anatomical region of the patient, wherein the images have been recorded during different examinations of the patient.
providing an output to a user, related to the medical scan images according to the result of the predefined relevance criteria, that comprises a workflow showing the order (e.g. a worklist in a RIS system shows patients in an order and a worklist in the PACS system shows the order of medical data for review. With the above invention changing the priority of the order of a patient’s x-rays for review, this performs the feature of showing a radiologist patient scans reading order that are impacted by the detected patient abnormalities, which is taught in ¶ [09], [39] and [50] above.).
However, Conjeti fails to specifically teach the features of outputs the characterised features and medical scan images sorted according to a predefined relevance criteria, providing an output to a user that comprises a workflow showing the order in which the plurality of medical scan images should be reviewed.
However, this is well known in the art as evidenced by Lyman. Similar to the primary reference, Lyman discloses performing a medical scan of a user (same field of endeavor or reasonably pertinent to the problem).
Lyman discloses receiving an input of a plurality of medical scans from a single scan for the single patient (e.g. the system includes one or more image slices that correspond to a single scan that can be evaluated and ordered, which is taught in ¶ [54].);
[0054] The medical scan image data 410 can include one or more images corresponding to a medical scan. The medical scan image data 410 can include one or more image slices 412, for example, corresponding to a single x-ray image, a plurality of cross-sectional, tomographic images of a scan such as a CT scan, or any plurality of images taken from the same or different point at the same or different angles. The medical scan image data 410 can also indicate an ordering of the one or more image slices 412. Herein, a “medical scan” can refer a full scan of any type represented by medical scan image data 410. Herein, an “image slice” can refer to one of a plurality of cross-sectional images of the medical scan image data 410, one of a plurality of images taken from different angles of the medical scan image data 410, and/or the single image of the medical scan image data 410 that includes only one image. Furthermore “plurality of image slices” can refer to all of the images of the associated medical scan, and refers to only a single image if the medical scan image data 410 includes only one image. Each image slice 412 can include a plurality of pixel values 414 mapped to each pixel of the image slice. Each pixel value can correspond to a density value, such as a Hounsfield value or other measure of density. Pixel values can also correspond to a grayscale value, a RGB (Red-Green-Blue) or other color value, or other data stored by each pixel of an image slice 412.
[0066] Display parameter data 470 can indicate parameters indicating an optimal or preferred display of the medical scan by an interactive interface 275 and/or formatted report for each abnormality and/or for the scan as a whole. Some or all of the display parameter data can have separate entries for each abnormality, for example, generated automatically by a subsystem 101 based on the abnormality annotation data 442. Display parameter data 470 can include interactive interface feature data 471, which can indicate one or more selected interface features associated with the display of abnormalities and/or display of the medical scan as a whole, and/or selected interface features associated with user interaction with a medical scan, for example, based on categorized interface feature performance score data 720 and a category associated with the abnormality and/or with the medical scan itself. The display parameter data can include a slice subset 472, which can indicate a selected subset of the plurality of image slices that includes a single image slice 412 or multiple image slices 412 of the medical scan image data 410 for display by a user interface. The display parameter data 470 can include slice order data 473 that indicates a selected custom ordering and/or ranking for the slice subset 472, or for all of the slices 412 of the medical scan. The display parameter data 470 can include slice cropping data 474 corresponding to some or all of the slice subset 472, or all of the image slices 412 of the medical scan, and can indicating a selected custom cropped region of each image slice 412 for display, or the same selected custom cropped region for the slice subset 472 or for all slices 412. The display parameter data can include density window data 475, which can indicate a selected custom density window for display of the medical scan as a whole, a selected custom density window for the slices subset 472, and/or selected custom density windows for each of the image slices 412 of the slice subset 472, and/or for each image slice 412 of the medical scan. The density window data 475 can indicate a selected upper density value cut off and a selected lower density value cut off, and/or can include a selected deterministic function to map each density value of a pixel to a grayscale value based on the preferred density window. The interactive interface feature data 471, slice subset 472, slice order data 473, slice cropping data 474, and/or the density window data 475 can be selected via user input and/or generated automatically by one or more subsystems 101, for example, based on the abnormality annotation data 442 and/or based on performance score data 720 of different interactive interface versions.
[0101] In various embodiments, the medical scan assisted review system 102 is operable to receive, via a network, a medical scan for review. Abnormality annotation data 442 is generated by identifying one or more of abnormalities in the medical scan by utilizing a computer vision model that is trained on a plurality of training medical scans. The abnormality annotation data 442 includes location data and classification data for each of the plurality of abnormalities and/or data that facilitates the visualization 825 of the abnormalities in the scan image data 410. Report data 830 including text describing each of the plurality of abnormalities is generated based on the abnormality data. The visualization 825 and the report data 830 (collectively displayed annotation data 820) is transmitted to a client device. A display device associated with the client device displays the visualization 825 in conjunction with the medical scan via an interactive interface, and the display device further displays the report data 830 via the interactive interface.
[0136] The displayed annotation data 820 can include report data 830 corresponding to each abnormality, or the entire scan, which can be automatically generated and/or retrieved for display to the user in conjunction with each abnormality, allowing the user to confirm and/or edit the report data 830. The report data 830 can include some or all of natural language text data 448 and/or report data 449 of the medical scan. Generating the report data 830 can include automatically generating text describing each abnormality based on location data and/or classifying data of the abnormality such as abnormality annotation data 442, for example, by utilizing a medical scan report generator function or other generator function trained on a training set of abnormality data or other diagnosis data and corresponding medical reports, annotations, or other natural language data or text describing the medical scan. Alternatively, some or all of the report data 830 can correspond to annotations and/or a report written by another user as natural language text, for example retrieved from the medical scan database 342. The report data 830 can be generated and/or retrieved for display by the medical scan assisted review system 102, in conjunction with initially displaying each abnormality and/or in response to generating the scan review data 810 for each abnormality based on user feedback. When report data 830 is presented to the user for review, the user can elect to accept the report data 830 as it stands based on user input to the interactive interface 275. The user can also elect to revise some or all of the report data 830, for example, where the natural language text of the report is presented in a text editor window and the user can type new text and/or delete existing text. The revisions entered by the user can be included in the scan review data 810, and can be used to modify diagnosis data 440 by extracting relevant terms from the revised natural language text, for example, by utilizing the medical scan natural language analysis function. Alternatively, when scan review data 810 is generated based on other user input indicating edits to the displayed annotation data, for example, if a user highlights a region with a new abnormality and/or modifies classifier data of an existing category based on a menu option, modified natural text data can be automatically generated in response to the scan review data 810 indicating that the displayed annotation data needs to be edited. Thus, original natural text data displayed in the report data 830, for example, in conjunction with initially indicating an abnormality, can be automatically changed to the modified natural language text in response to generating the scan review data 810, where the displayed report data 830 automatically reflects the modified natural language text.
outputs the characterised features and medical scan images sorted according to a predefined relevance criteria, providing an output to a user that comprises a workflow showing the order in which the plurality of medical scan images from the single scan should be reviewed (e.g. a subsystem can be used to assign scores regarding priority or malignancy to a medical scan. The medical scans seen within a queue can be automatically ordered based on the priority score that can be based on a malignancy score, which is taught in ¶ [156]-[160]. The scans can be seen by a reviewer that shows a nodule or disease, which the nodule is taught in ¶ [152]. The system used to present the queue for reviewing scans in a particular order is a part of a machine learning model, which is taught in ¶ [259] and [260]. This reference in combination with the prior reference discloses showing an order of scans that comprised a disease performed automatically to show the reviewer a list for review in an automated manner. In ¶ [106], [129] and [130], slices of a medical scan can be ordered in a manner that is viewed based on how these slices relate to a similar abnormality area captured within a scan. In ¶ [259]-[261], [265]-[268] discloses creating a subset of slices that are selected including an abnormality within the center as a first image slice to be displayed. The system further orders areas around the abnormality to be shown or reviewed, so that as the user goes through the slices, the user is shown the workflow through how the images around an abnormality have been arranged for review based on their assigned probability. Images that have no or zero probability are not included in a set of reviewed slices, which saves review time.).
[0152] In FIG. 8N, the interactive interface enters the new abnormality mode in response to the scan review data 810 indicating that the user elects to enter a new finding. The user can type the report entry corresponding to the new finding in a text window as shown in FIG. 8O. In FIG. 8P, the user elects to select the region of interest corresponding to the new finding, such as a nodule that was overlooked by the medical scan assisted review system, and the interactive interface 275 prompts the user to indicate the region of interest by selecting points on the image slice corresponding to vertices of a polygon that surrounds the nodule. In FIG. 8Q, the interactive interface displays five vertices 850 selected by the user, and the interactive interface 275 prompts the user to indicate when they have finished by double clicking. The medical scan assisted review system 102 automatically generates scan review data 810 corresponding to the new finding based on the text entered by the user in the text window and the polygon indicating the region of interest, automatically determined based on the five vertices 850 indicated by the user, as presented in the interactive interface 275 of FIG. 8R in response to the user electing to approve the new finding, and this scan review data 810 corresponding to the new finding can be added to the medical scan database and/or utilized in training sets used to improve the performance of the medical scan assisted review system 102 or other subsystems in subsequent uses.
[0156] A plurality of triaged and/or uploaded scans can be queued for review by a selected user, and the medical scan assisted review system 102 can present a listed queue of scans for review via another view of the interactive interface 275 presented by the display device. This can be based on a plurality of medical scans triaged to the user and/or in response to receiving a plurality of medical scans uploaded to the medical scan processing system 100 by user. The queue of scans can be displayed as line item data in a row, and corresponding data can be displayed such as include patient data, risk factor data, priority score data, a diagnosis summary, abnormality classifier data, confidence score data, or other data retrieved from the medical scan database 342 or generated by one or more subsystems 101.
[0157] The queue of scans can be displayed via the interactive interface 275 in an order based on an automatically generated priority score or a priority score retrieved from scan priority data 427 of the medical scan database 342. For example, the priority score can be based on a manually assigned score for an incoming scan, based on the date of the scan, based on a severity of patient symptoms, severity of previous diagnosis data of the scan, or severity of the diagnosis data 440 automatically generated for the medical scan, for example, based on a malignancy score indicated in diagnosis data 440. The user can re-sort and/or filter the queue based on one or more selected criteria, selected via user input to the interactive interface and/or determined based on queue criteria preferences associated with the user, for example, mapped to user profile entry 354. The selected criteria can include such as selecting to view the filtered list of scans based on criteria for one or more selected abnormality pattern types. For example, the user can select to view the filtered list of scans where an abnormality pattern corresponding to cardiomegaly was detected, and can further select to sort the list of scans in reverse order by a confidence score corresponding to detection of cardiomegaly of confidence score data 460. As another example, consider the case where diagnosis data 440 has multiple entries corresponding to multiple diagnosis authors, for example with a first entry was generated based on user input by another user of the system and a second entry generated automatically by utilizing a medical scan image analysis function. The user can select to view the filtered list of scans where an abnormality was detected by the first diagnosis author that was not reported by the second diagnosis author, or vice versa. This can be used to quickly find discrepancies between known diagnosis data and other diagnosis data generated by a subsystem, for example, by utilizing a medical scan analysis function. The user can continue to re-sort and/or further filter or un-filter the queue by adding or removing sorting and/or filter criteria.
[0158] The user can be automatically presented a medical scan for review from the top of a sorted and/or filtered queue, or the original queue, by the medical scan assisted review system 102 as described herein. Alternatively or in addition, the user can select a scan from the original queue of scans, or the re-sorted and/or filtered queue of scans, based on clicking the corresponding row of the selected scan or other input to the interactive interface, and the selected medical scan will be presented by the medical scan assisted review system 102 as described herein. The medical scan review system can return to the displayed queue of scans after a user has completed review of the current scan. The user can elect to re-order and/or re-filter the queue by providing new criteria, such as selecting a new confidence score threshold or selecting new abnormality pattern type. The user can also elect to select a new medical scan for review from the displayed queue. In other embodiments, medical scan review system can automatically display the next scan in the queue once the review of the current scan is complete, without returning the view of listed scans.
[0159] In some embodiments, the user can elect to confirm the diagnosis data 440 for a selected medical scan without viewing the medical scan. For example, where the user can elect to confirm the diagnosis data 440 by selecting a menu option presented for each scan in the list of scans via the interactive interface 275. This can be based on confidence score data 460 displayed as a line item in conjunction with the medical scan, for example, where the user elects to confirm diagnosis data 440 without viewing the scan because the displayed confidence score is 99%. The user can select auto-confirm criteria, such as “automatically confirm all normal scans” or “automatically confirm all diagnosis data that indicates cardiomegaly with a confidence score that is greater than 90%”. In such embodiments, the queue can be automatically filtered based on the auto-confirm criteria, where automatically confirmed medical scans are not listed. The auto-confirm criteria can be selected via user input to the interactive interface 275 and/or determined based on queue criteria preferences associated with the user, for example, mapped to user profile entry 354. In some embodiments, the auto-confirm criteria will be learned and selected by a subsystem and/or the auto-confirm criteria will be entered by an administrator.
[0160] In various embodiments, the medical scan assisted review system 102 will only display medical scans with detected abnormalities, for example, where normal scans are included in the auto-confirm criteria. For example, medical scans automatically determined to be normal with at least a threshold confidence score will not be presented to the user for review, and will automatically be filtered from the queue. Diagnosis data, report data, and/or a flag indicating the scan is normal will automatically be mapped to the medical scan in the medical scan database and/or transmitted to the responsible medical entity without user intervention. For quality control, a threshold proportion of normal scans, or other auto-confirmed scans based on other criteria, can be randomly or psuedo-randomly selected and presented to the user for review.
[0106] The interactive interface 275 can present a medical scan to a user by displaying medical scan image data 410. This can include displaying a single image slice 412 of the medical scan and/or displaying a selected number or automatically determined number of image slices 412 simultaneously in adjacent views. If the medical scan only includes a single image, the single image can be displayed. If multiple image slices are available, the interactive interface 275 can allow a user to scroll through ordered slice images of the medical scan, where the image slices are displayed one at a time sequentially in a forwards or backwards direction based on user input to move a scroll bar displayed by the interface, user input to arrows presented on a keyboard of the client device, user input to a scroll wheel on a mouse of the client device, or other user input indicating the user elects to navigate sequentially forwards or backwards through the slice images. The interactive interface can allow a user to jump to particular slices of the medical scan, for example, where a slice is selected via user input to the interactive interface based on a slice number, thumbnail image, a point selected on a scroll bar, or other indicator. The interactive user interface can also allow a user to zoom in on a selected region of a particular slice, display multiple slices at the same time, or otherwise navigate the medical scan data visually based on selecting menu options presented by the interactive user interface or based on other user input.
[0129] In various embodiments, a first window displaying an image slice 412 of the medical scan and an adjacent second window displaying an image slice of a similar scan will display image slices 412 corresponding to the same image slice number in the sequence, for example, where the image slice 10 out of 30 is displayed for the medical scan in the first window, and image slice 10 out of 30 is displayed for the second medical scan in the second window. In various embodiments, the displayed slice of the similar scan can be selected by automatically determining the appropriate anatomical region corresponding to the currently displayed slice of the current scan. For example, the medical scan assisted review system can automatically select the appropriate corresponding image slice in the previous scan based on the similar scan display parameter data 483, by determining which image slice of the previous scan compares most favorably to the currently displayed image slice of the current scan based on performing a medical scan similarity analysis function on the image slices as described herein, by determining the outline of an anatomical region of the slice and determining a slice in the previous scan that most closely matches the anatomical region borders, and/or by another visual comparison. In such embodiments, the user can use a single scroll bar or other single user input indication to jump to a different image slice, and the multiple windows can simultaneously display the same numbered image slice or can scroll or jump by the same number of slices if different slice numbers are initially displayed. In some embodiments, three or more adjacent windows corresponding to the medical scan and two or more similar scans are displayed, and can all be controlled with the single scroll bar in a similar fashion.
[0130] In other embodiments, the selected slice of the similar medical scan in the second window is a single or subset of slice images identified in the set, is automatically selected based on identifying a slice image of the similar scan that includes a most similar view of the abnormality as the currently displayed slice for the current scan. For example, slice 12 of the medical image includes a view of the abnormality that is most similar to slice 31 of the similar medical image, and slice 31 will automatically be selected for display in conjunction with the user selecting to view slice 12. The user can then switch to view slice 13 of the medical image that includes a different view of the abnormality that is most similar to slice 35 of the similar medical image, and slice 35 will automatically be selected for display in conjunction with the user scrolling or jumping from slice 12 to slice 13 of the medical image. In other embodiments, the user can scroll or otherwise navigate through slices for the medical scan and one or more similar medical scans separately.
[0253] FIG. 13 presents an embodiment of a medical scan image analysis system 112. A medical scan image analysis system 112 can be used to generate and/or perform one or more medical scan image analysis functions by utilizing a computer vision-based learning algorithm on a training set of medical scans with known annotation data, diagnosis data, labeling and/or medical code data, report data, patient history data, patient risk factor data, and/or other metadata associated with medical scans. These medical scan image analysis functions can be used to generate inference data for new medical scans that are triaged or otherwise require inferred annotation data, diagnosis data, labeling and/or medical code data, and/or report data. For example, some medical scan image analysis functions can correspond to medical scan inference functions of the medical scan diagnosing system or other medical scan analysis functions of the medical scan analysis function database 348. The medical scan image analysis functions can be used to determine whether or not a medical scan is normal, to detect the location of an abnormality in one or more slices of a medical scan, and/or to characterize a detected abnormality. The medical scan image analysis system can be used to generate and/or perform computer vision based medical scan image analysis functions utilized by other subsystems of the medical scan processing system as described herein, aiding medical professionals to diagnose patients and/or to generate further data and models to characterize medical scans. The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.
[0259] In embodiments where the medical scan image analysis system is used in conjunction with the medical scan diagnosing system, each of the medical scan image analysis functions associated with each neural network model can correspond to one of the plurality of neural network models generated by the medical scan image analysis system. For example, each of the plurality of neural network models can be trained on a training set classified on scan type, scan human body location, hospital or other originating entity data, machine model data, machine calibration data, contrast agent data, geographic region data, and/or other scan classifying data as discussed in conjunction with the medical scan diagnosing system. In embodiments where the training set classifiers are learned, the medical scan diagnosing system can determine which of the medical scan image analysis functions should be applied based on the learned classifying criteria used to segregate the original training set.
[0260] A computer vision-based learning algorithm used to create each neural network model can include selecting a three-dimensional subregion 1310 for each medical scan in the training set. This three-dimensional subregion 1310 can correspond to a region that is “sampled” from the entire scan that may represent a small fraction of the entire scan. Recall that a medical scan can include a plurality of ordered cross-sectional image slices. Selecting a three-dimensional subregion 1310 can be accomplished by selecting a proper image slice subset 1320 of the plurality of cross-sectional image slices from each of the plurality of medical scans, and by further selecting a two-dimensional subregion 1330 from each of the selected subset of cross-sectional image slices of the each of the medical scans. In some embodiments, the selected image slices can include one or more non-consecutive image slices and thus a plurality of disconnected three-dimensional subregions will be created. In other embodiments, the selected proper subset of the plurality of image slices correspond to a set of consecutive image slices, as to ensure that a single, connected three-dimensional subregion is selected.
[0261] Consider the case where the selected image slices correspond to a set of consecutive image slices. Selecting the two-dimensional subregions 1330 from each image slice in the of a scan can include selecting the same region of pixels or a region determined from same vertical and horizontal borders to generate a three-dimensional subregion that corresponds to a prism shape. For example, selecting the same-sized square region centered at the same (x,y) coordinate pair of each of the image slices of a scan would result in a three-dimensional subregion that corresponds to a rectangular prism, and selecting the same-sized circular region centered at the same (x, y) coordinate pair of each of the image slices would result in a three-dimensional subregion that corresponds to a cylindrical shape. In other embodiments, different sized and/or shaped two-dimensional subregions can be selected from each image slice of the scan. For example, different sized circular regions can be selected that are centered at the same (x,y) coordinate pair of each of the image slices, where the largest circle is selected from the center image slice in the set of ordered consecutive slices, where the smallest circles are selected from the first and last image slices in the set of ordered consecutive slices, where the circles selected from the first slice to the center slice monotonically increase in size, and where the circles selected from the center slice to the last slice monotonically decrease in size. Such as strategy would result in a three-dimensional subregion that resembles a sphere.
[0265] In various embodiments, a probability distribution function (PDF) can be used to determine the center location of the subregion, where selecting a subregion that is closer to the center to the subregion is more probable than selecting a subregion that is further from the center of the subregion. FIG. 13B presents an example embodiment of utilizing an x-axis PDF 1342, centered at a central x coordinate of the abnormality 1340, to select an x coordinate, and utilizing a y-axis PDF 1344, centered at a central y coordinate of the abnormality 1340, to select a y coordinate, where the selected x and y coordinates are used to determine the location of the subregion. In this fashion, consider a first possible subregion centered at a first location that is a first distance away from a center or nearest border of the known abnormality, and a second possible subregion centered at a second location that is a second distance away from the center or nearest border of the known abnormality. The PDF could dictate that selecting the first possible subregion is more probable than selecting the second possible subregion as a result of the first distance being shorter than the second distance. The two-dimensional subregions can be selected based on the ideal two-dimensional subregion size and/or shape requirements and the selected center (x,y) coordinate pair. In embodiments where the ideal two-dimensional subregion size requirements includes a size range, the subregion will further be determined based on probabilities assigned to each of the sizes in the range. In embodiments where the ideal two-dimensional subregion shape requirements includes a set of shapes and/or a set of shape parameters, the subregion will further be determined based on based on probabilities assigned to each of the shapes in the set of shapes and/or probabilities assigned to values of each parameter in the set of shape parameters.
[0266] Similarly, as illustrated in FIG. 13C, a center image slice of the selected proper image slice subset 1320 can be based on a slice PDF 1346, for example, centered about a central slice that includes the known abnormality in the plurality of ordered image slices. The remainder of the image slices will be determined based on the ideal slice number requirements and the center image slice. In embodiments where the ideal slice number requirement includes a range, the remainder of the image slices can be further determined based on probabilities assigned to each number of slices in the ideal slice number range or based on other deterministic factors.
[0267] Each PDF, such as PDFs 1342, 1344, and 1346, can be based on a Gaussian distribution or other distribution centered at the location of the abnormality with three random or psuedo-random variables: the center of the consecutive subset of image slices and the center (x,y) coordinate pair for the two-dimensional subregions of each image slice. In various embodiments, the PDF model and/or model parameters are determined based on received PDF data. For example, the received PDF parameter data can include the PDF itself, can indicate that a Gaussian distribution with a center slice variance parameter of 3 slices, with an x variance parameter of 10 pixels, and a y variance parameter of 10 pixels. The PDF model and/or parameters of a selected PDF model can also be automatically generated by the medical scan image analysis system, for example, after testing models generated utilizing different PDFs to determine the model that produces the most accurate inferences.
[0268] In some embodiments, all possible subregion locations are assigned a non-zero probability. In other embodiments, some locations are assigned a probability of zero, and thus subregions corresponding to such locations will not be selected. For example, all subregions at locations that do not include the abnormality are assigned a probability of zero and are never sampled. In some embodiments, subregions at locations that do not correspond to a region of interest, for example, locations that do not correspond to a location of the human body corresponding to the neural network model to be created can be assigned a probability of zero. For example, if the training set is created to generate a neural network model for detecting abnormalities in the lung, regions of each cross-sectional image slice that do not include the lung, or entire slices for example at the beginning or end of the scan that do not include the lung, can be assigned a probability of zero. This masking step can include determining regions of the scan that do not correspond to the region of interest, for example, by applying another computer vision-based masking function. The computer vision-based masking function can be generated automatically by the medical scan image analysis system or other system of the medical scan processing system, for example, where the computer vision-based masking function is trained on the training data, for example, where regions to be masked are identified manually in the training data.
Therefore, in view of Lyman, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of receiving an input of a plurality of medical scans from a single scan for the single patient; outputs the characterised features and medical scan images sorted according to a predefined relevance criteria, providing an output to a user that comprises a workflow showing the order in which the plurality of medical scan images from the single scan should be reviewed, incorporated in the device of Conjeti, in order to sort scans with signs of severe disease for review in a queue for a reviewer in a specific order, which can aid the medical professional in reviewing medical scans (as stated in Lyman ¶ [100]).
Re claim 2: However, Conjeti fails to specifically teach the features of the method of claim 1, wherein the output is at least one of: a display of medical images sorted according to the predefined relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria.
However, this is well known in the art as evidenced by Lyman. Similar to the primary reference, Lyman discloses performing a medical scan of a user (same field of endeavor or reasonably pertinent to the problem).
Lyman discloses wherein the output is at least one of: a display of medical images sorted according to the predefined relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria (e.g. an output of the system can consist of a display of medical images that can be previewed or displayed for review or a queue showing the potential reports for review, which is taught in ¶ [152] and [156]-[160] above.).
Therefore, in view of Lyman, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the output is at least one of: a display of medical images sorted according to the predefined relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria, incorporated in the device of Conjeti, in order to sort scans with signs of severe disease for review in a queue for a reviewer in a specific order, which can aid the medical professional in reviewing medical scans (as stated in Lyman ¶ [100]).
Re claim 3: Conjeti discloses the method of claim 1, further comprising the steps of:
detecting one or more features of the plurality of medical scan images (e.g. an abnormality is detected in the scan in order to compare this abnormality to another scan, which is taught in ¶ [39] above); and
characterising the one or more detected features from the plurality of medical scan images; before providing the input to the optimisation model (e.g. the abnormalities can be determined by examination unit before the abnormalities are sent to the determination unit, which is performed by a machine learning algorithm. This is taught in ¶ [60]-[62] and [69] above.).
Re claim 4: Conjeti discloses the method of claim 3, wherein the optimisation model outputs the N most relevant features according to the predefined relevance criteria, where N is a configurable parameter of the method (e.g. a threshold is used to determine if an abnormality should trigger an alert. This threshold used can be defined for actionable changes and can be defined in different ways, which is taught in ¶ [18]-[21]. The optimisation model allows for a flag of the images to be performed associated with the abnormalities, which is taught in ¶ [39] above.).
[0018] The threshold should be defined such that an actionable change that is complying with Standard of Care (or Clinical Standard Operating Procedures) is indicated. The Standard of Care as well as an actionable change is well known in the art concerning a predefined disease. It should be noted that an actionable change is not any change per say, but only ones that trigger changes to patient management. It should also be noted that the method according to the invention is not anticipating the examination of a physician, but may use the thresholds defined for actionable changes.
[0019] For automatically determining changes, said decision algorithm is used, that is designed to calculate whether changes, e.g. in the spatial extent (dimensions) and/or density, texture, shape of the abnormality manifest as an actionable change that is complying with Standard of care. This decision algorithm is necessary. It can be designed such that it is calculating whether a numerical value of a change exceeds a threshold in a very simple embodiment. However, it also could be a machine learning (especially deep learning) algorithm that is specially trained to measure changes of abnormalities in images. Such algorithm facilitates assessment of series of images that are indicative of clinically actionable changes. It should be noted that this decision algorithm should preferably be invoked automatically before a clinician reviews the case, especially in the case of triage.
[0020] The threshold may be a value (e.g. a volume, measurement) or a predefined region that should not be exceeded. However, the threshold may also be a limit for the measure of the change itself, e.g. a threshold for the first derivative of the change with respect to time.
[0021] In the case the assessed changes exceed the predefined threshold:
triggering an alert notification and/or change a worklist prioritization concerning the respective patient.
Re claim 5: Conjeti discloses the method of claim 1, wherein the plurality of medical scan images is one of: CT scan, PET scan, MRI scan, SPECT scan (e.g. CT images can be captured, which is taught in ¶ [10].).
[0010] It is clear that both images should show the same region of interest, e.g. the same organ, since this region is examined for changes. It is also clear that the images should be recorded during different examinations of the patient, since a longitudinal examination is intended, i.e. an examination over two or more time points, e.g. at different hours, days, weeks or months, possibly at a different (physical) location. The more images are processed, the more timepoints are available. Any further image can be processed with this method with respect to the temporally adjacent image. Although it is preferred that both images are of the same type, e.g. both images are X-ray images or CT-images (CT: computer tomography), this is not in every case necessary. Instead of the term "image" also the term "medical image" can be used. For example, the image can be an object according to the DICOM (acronym for "Digital Imaging and Communications in Medicine") standard. examining the images for abnormalities of a predetermined type.
Re claim 6: Conjeti discloses the method of claim 5, wherein the plurality of medical scan images are 2D images or 3D images (e.g. 2D or 3D images can be captured, which is taught in ¶ [55].).
[0055] It should be noted that the images could be 2D-images or 3D-images. By using deep learning algorithms, the images do not have to be reconstructed images, but could also comprise raw data. In general, it is preferred that the images are DICOM-datasets.
Re claim 7: Conjeti discloses the method of claim 5, wherein the one or more medical scan images are images showing all or part of a lung (e.g. a lung area is captured with the scan, which is taught in ¶ [39] above.), and
the characterised features are one or more of: lung nodule, features indicating the presence of at least one of emphysema, fibrotic tissue, consolidation and scarring (e.g. a lung lesion or evidence of emphysema can be detected as an abnormality, which is taught in ¶ [46].).
[0046] According to a preferred embodiment, an abnormality is a (radiographic) finding of the group comprising malpositioned lines or tubes of immediate clinical concern, tension pneumothoraces, pulmonary emboli, lung lesions with high possibility of being active tuberculosis, superior vena cava occlusion, large pericardial effusion and/or suspected tamponade of any cause, active post-traumatic hemorrhage, tracheal obstruction, lobar or lung collapse, pneumomediastinum, interstitial emphysema, extensive subcutaneous emphysema, pleural effusion, significant vena cava compression and pneumonia, or a non-thoracic finding from the group comprising suspected nonaccidental trauma, malpositioned line or tube of immediate clinical concern (e.g., ET tube or enteric tube in bronchus), allergic reaction or other adverse event, foreign body with potential immediate and/or severe consequences, intracranial or spinal hemorrhage (parenchymal, subarachnoid, subdural, epidural), non-hemorrhagic stroke or suspected stroke, thrombolytic candidate, intracranial mass with significant mass effect (midline shift/herniation/hydrocephalus), brain herniation, symptomatic hydrocephalus (malfunctioning shunt or new diagnosis of any cause), depressed skull fracture, post-traumatic pneumocephalus, arterial dissection, brain death (nuclear study or other), severe spinal cord compression of any cause, unstable spine fracture, cord hemorrhage or infarct, airway obstruction, unexplained pneumoperitoneum, closed loop intestinal obstruction, intestinal ischemia and/or portal/mesenteric, venous gas, pseudoaneurysm or active hemorrhage (post-trauma, GI bleed, other), high-grade intra-abdominal organ injury, testicular torsion, ovarian torsion, ectopic pregnancy (high likelihood), placental abruption, uterine rupture, high grade kidney injury and/or ureteral or bladder injury post-trauma, non-spinal fracture and/or dislocation with risk of vascular compromise, necrotizing fasciitis, Ruptured/leaking arterial aneurysm (thoracic or abdominal aortic or other), limb-threatening arterial or venous occlusion or high-grade stenosis, arterial dissection and intramural hematoma (aortic or other).
Re claim 8: Conjeti discloses the method of claim 3, wherein the detected features are characterised according to one or more of: a malignancy score, an invasiveness score, a feature size (e.g. the size of area is considered for the abnormality detection, which is taught in ¶ [13].).
[0013] These changes could be changes of dimensions, e.g. the progression of the collapse of the lung concerning a pneumothorax or the expansion of a haemorrhage. For example, dimensional changes could be computed with area-comparison or apex to cupola distance comparison. However, changes could also be changes of opacity, texture and/or shape of an anatomical region. Concerning digital images, a "change" is a change in digital image information of the predefined region. Since the image information is usually the grayscale value of the pixels of the image, the change is the change of grayscale and/or size and/or form of image-elements in the predefined region.
Re claim 15: Conjeti discloses a device for analysing medical scan images from a single patient comprising:
one or more processors (e.g. processors are utilized, which is taught in ¶ [30] and [31].) configured to:
[0030] Some units or modules of the determination system mentioned above can be completely or partially realized as software modules running on a processor of a computing system. A realization largely in the form of software modules can have the advantage that applications already installed on an existing computing system can be updated, with relatively little effort, to install and run these units of the present application. The object of the invention is also achieved by a computer program product with a computer program that is directly loadable into the memory of a computing system, and which comprises program units to perform the steps of the inventive method when the program is executed by the computing system. In addition to the computer program, such a computer program product can also comprise further parts such as documentation and/or additional components, also hardware components such as a hardware key (dongle etc.) to facilitate access to the software.
[0031] A computer readable medium such as a memory stick, a hard-disk or other transportable or permanently-installed carrier can serve to transport and/or to store the executable parts of the computer program product so that these can be read from a processor unit of a computing system. A processor unit can comprise one or more microprocessors or their equivalents.
receive an input of a plurality of medical scan images for the single patient (e.g. the invention discloses acquiring multiple X-ray images of a patient, which is taught in ¶ [39].):
provide an input of one or more characterised features from the plurality of medical scan images, in addition to the medical scan images to an optimisation model (e.g. a machine learning algorithm is used to evaluate first and second images to determine the abnormalities present within the images. The abnormalities found within the deep machine learning model can be considered as input into the device, which is taught in ¶ [39] above. In addition, the abnormalities are passed to the determination unit with the images in order to determine changes of the abnormality. The determination unit uses a decision algorithm to perform this feature, which is taught in ¶ [60]-[62] and [69] above.),
where the optimisation model outputs the features and images (e.g. the invention is directed towards organizing the reading worklist of the chest x-rays of a patient over time and changing this patient’s examination order in the workflow based on the detected abnormalities, which is taught in ¶ [09] and [39] above.); and
providing an output related to the medical images to a user according to a result of the predefined relevance criteria, that comprises a workflow showing the order (e.g. a worklist in a RIS system shows patients in an order and a worklist in the PACS system shows the order of medical data for review. With the above invention changing the priority of the order of a patient’s x-rays for review, this performs the feature of showing a radiologist patient scans reading order that are impacted by the detected patient abnormalities, which is taught in ¶ [09], [39] and [50] above.).
However, Conjeti fails to specifically teach the features of outputs the characterised features and images sorted according to a predefined relevance criteria, providing an output related to the medical scan images to a user that comprises a workflow showing an order in which the plurality of medical scan images should be reviewed.
However, this is well known in the art as evidenced by Lyman. Similar to the primary reference, Lyman discloses performing a medical scan of a user (same field of endeavor or reasonably pertinent to the problem).
Lyman discloses receive an input of a plurality of medical scan images from a single scan for the single patient (e.g. the system includes one or more image slices that correspond to a single scan that can be evaluated and ordered, which is taught in ¶ [54] above.);
outputs the characterised features and images sorted according to a predefined relevance criteria, providing an output that comprises a workflow showing an order in which the plurality of medical scan images from the single scan should be reviewed (e.g. a subsystem can be used to assign scores regarding priority or malignancy to a medical scan. The medical scans seen within a queue can be automatically ordered based on the priority score that can be based on a malignancy score, which is taught in ¶ [156]-[160] above. The scans can be seen by a reviewer that shows a nodule or disease, which the nodule is taught in ¶ [152] above. The system used to present the queue for reviewing scans in a particular order is a part of a machine learning model, which is taught in ¶ [259] and [260] above. This reference in combination with the prior reference discloses showing an order of scans that comprised a disease performed automatically to show the reviewer a list for review in an automated manner. In ¶ [106], [129] and [130] above, slices of a medical scan can be ordered in a manner that is viewed based on how these slices relate to a similar abnormality area captured within a scan. In ¶ [259]-[261], [265]-[268] above discloses creating a subset of slices that are selected including an abnormality within the center as a first image slice to be displayed. The system further orders areas around the abnormality to be shown or reviewed, so that as the user goes through the slices, the user is shown the workflow through how the images around an abnormality have been arranged for review based on their assigned probability. Images that have no or zero probability are not included in a set of reviewed slices, which saves review time.).
Therefore, in view of Lyman, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of receive an input of a plurality of medical scan images from a single scan for the single patient; outputs the characterised features and images sorted according to a predefined relevance criteria, providing an output that comprises a workflow showing an order in which the plurality of medical scan images from the single scan should be reviewed, incorporated in the device of Conjeti, in order to sort scans with signs of severe disease for review in a queue for a reviewer in a specific order, which can aid the medical professional in reviewing medical scans (as stated in Lyman ¶ [100]).
Re claim 16: Conjeti discloses the device for analyzing medical scan images from a single patient of claim 15, wherein the plurality of medical scan images is one of: CT scan, PET scan, MRI scan or SPECT scan (e.g. CT images can be captured, which is taught in ¶ [10] above.), and the one or more medical scan images are images showing all or part of a lung (e.g. a lung area is captured with the scan, which is taught in ¶ [39] above.).
Re claim 17: Conjeti discloses the device for analyzing medical scan images from a single patient of claim 15, wherein the one or more processors are further configured to detect multiple classes of features in the plurality of input medical scan images (e.g. the features detected can be lines sensed within an image or a size of an area that is compared to other scans, which are separate classes and explained in ¶ [46]-[48] above.).
Re claim 18: Conjeti discloses the device for analyzing medical scan images from a single patient of claim 17, wherein the one or more processors are further configured to characterised the one or more detected features from the plurality of medical scan image, before the detected features are provided to the optimisation model (e.g. the abnormalities can be determined by examination unit before the abnormalities are sent to the determination unit, which is performed by a machine learning algorithm. This is taught in ¶ [60]-[62] and [69] above.).
Re claim 20: However, Conjeti fails to specifically teach the features of the device for analyzing medical scan images from a single patient of claim 15, wherein the output is at least one of: a display of medical scan images sorted according to the relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria.
However, this is well known in the art as evidenced by Lyman. Similar to the primary reference, Lyman discloses performing a medical scan of a user (same field of endeavor or reasonably pertinent to the problem).
Lyman discloses wherein the output is at least one of: a display of medical scan images sorted according to the relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria (e.g. an output of the system can consist of a display of medical images that can be previewed or displayed for review or a queue showing the potential reports for review, which is taught in ¶ [152] and [156]-[160] above.).
Therefore, in view of Lyman, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the output is at least one of: a display of medical scan images sorted according to the relevance criteria, a report containing information about the medical images sorted according to the predefined relevance criteria, incorporated in the device of Conjeti, in order to sort scans with signs of severe disease for review in a queue for a reviewer in a specific order, which can aid the medical professional in reviewing medical scans (as stated in Lyman ¶ [100]).
Claim(s) 9-14 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Conjeti, as modified by Lyman, as applied to claims 3 and 17 above, and further in view of Allen (US Pub 2020/0160032).
Re claim 9: However, Conjeti fails to specifically teach the features of the method of claim 3, wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model.
However, this is well known in the art as evidenced by Allen. Similar to the primary reference, Allen discloses a machine learning model that determines features in an image (same field of endeavor or reasonably pertinent to the problem).
Allen discloses wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model (e.g. the reference discloses using two machine learning models to perform different parts of the analysis. A first tier can be used to determine the features while a second tier is used to characterize what the scores associated with the features represent, which is taught in ¶ [47] and [55].).
[0047] In some embodiments, the methods or processes described herein generally involve whole slide classification using clinically-relevant cellular features, such as cytomorphologic or histologic criteria. The methods use deep learning to predict whole slide-level pathology diagnoses based on aggregated cellular features chosen from, e.g., established guidelines or clinical practice. Deep learning is often considered an uninterpretable black box process. There are a number of approaches for explaining the behavior of deep learning-based machine learning models, but these are nearly all retrospective and limited. In order to take advantage of deep learning while maintaining interpretability, the approach described herein includes a separate deep learning-based sub-model trained for each cellular feature deemed relevant for a particular interpretation by clinical guidelines, providing a first tier of analysis. The results of these sub-models are then aggregated for all cells or tissue across the whole slide image using statistical summary measures (e.g. mean, standard deviation, skew). These statistical measures, as well as, in some embodiments, other additional data, can then be used in a second tier of machine learning analysis to predict a whole slide-level diagnosis, such as the presence or absence of a disease or disease type. This method is not only accurate, but also enables display of the reasoning for the diagnosis at the cellular and whole slide levels, and allows for optimization of outputs as well as optimization of the system models.
[0055] At stage 191, the system can generate an array of feature scores. The scores can be based on training received by the system. For example, a system can be trained against a pathologist's scores for each selected features, based on the particular representative feature and disease, condition, or process being examined. For example, a series of trained models can be used to produce an array of feature scores including scores for each of the selected features. One skilled in the art will appreciate that any of the various methods available for training one or more models for the systems and methods described herein may be used. In some embodiments, the systems, methods, and devices described herein can use deep learning. In some embodiments, the systems, methods, and devices described herein do not use simple end-to-end learning to analyze a slide image. In some embodiments, the systems, methods, and devices described herein use a first tier of trained models (e.g., a series of models including one or more models for each of two or more features), combined with a second tier trained model that processes an array of feature scores obtained from the first tier of models. In some embodiments, the systems, methods, and devices described herein use machine learning to perform one or more of classification, regression, clustering, and association. In some embodiments, the systems, methods, and devices described herein can employ one or more of dynamic time warping (DTW), decision trees, linear regression, neural networks, multinomial LR, Naive Bayes (NB), trained Gaussian NB, NB with dynamic time warping, MLR, Shannon entropy, support vector machine (SVM), one versus one support vector machine, k-means clustering, Q-learning, temporal difference (TD), deep adversarial networks, and the like. In some embodiments, the systems and methods described herein use one or more multiple instance learning models.
Therefore, in view of Allen, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model, incorporated in the device of Conjeti, in order to utilize models to perform the detection of features and characterizing of the features, which using automated analyses of biologic specimens can decrease time and costs of analyses when compared to traditional methods (as stated in Allen ¶ [14]).
Re claim 10: Conjeti discloses the method of claim 9, wherein the first and second machine learning model uses a neural network, a support vector machine or a random forest algorithm (e.g. leaning models are used for segmenting and determining abnormalities. A neural network is used for the models, which is taught in ¶ [39] above and [42].).
[0042] According to a preferred embodiment, a machine learning algorithm is used for examining the images and/or for determining changes of an abnormality and/or for determining whether the assessed changes exceed a predefined threshold. This machine learning algorithm is preferably a deep-learning network, preferably a deep belief network, ResNet, DenseNet, Autoencoder, capsule network, generative adversarial network, Siamese network, convolutional neural network, deep reinforcement learning algorithm, or preferably based on a machine learning technique of the group support vector machine, Bayesian model, decision tree and k-means clustering.
Re claim 11: Conjeti discloses the method of claim 10, wherein the optimization model uses a machine learning model such as a neural network (e.g. a trained machine learning algorithm is used to perform the feature of the optimization model, which is taught in ¶ [69] above. In addition other models can be used to serve as the optimization model, which is taught in ¶ [42] above.).
Re claim 12: Conjeti discloses the method of claim 9, wherein the feature detection model is a single model that detects multiple feature classes simultaneously (e.g. the features detected can be lines sensed within an image or a size of an area that is compared to other scans, which are separate classes and explained in ¶ [46] above, [47] and [48].).
[0047] According to a preferred embodiment, in the course of comparing the areas of an abnormality in the images, the speed of growth is determined, preferably based on the time difference between the recording of images, especially by determining the difference of the area between the images relative to the time between the recording of the images. In particular, the speed of growth is the speed of growth of the areas of the abnormality in the image. This time dependence (e.g. a slope of a curve) has the advantage that such dependence (e.g. the slope) provides a measure about how fast the change occurred.
[0048] In general, the images may provide information about the time point of their recording and the difference between the time of recording of the first image and the second image is preferably calculated together with the changes. The changes are then preferably compared with the threshold based on this time difference, so that e.g. a faster growth exceeds the threshold wherein a slower growth of the same size doesn't.
Re claim 13: However, Conjeti fails to specifically teach the features of the method of claim 9, wherein the feature detection model comprises a plurality of models, each model detecting a certain class of feature.
However, this is well known in the art as evidenced by Allen. Similar to the primary reference, Allen discloses a machine learning model that determines features in an image (same field of endeavor or reasonably pertinent to the problem).
Allen discloses wherein the feature detection model comprises a plurality of models, each model detecting a certain class of feature (e.g. the invention discloses having a model used to determine the feature and feature scores for each feature, which is taught in ¶ [47] and [55] above. Separate models can be used to recognize and produce values associated with different classes of features, which is taught in ¶ [99].).
[0099] In scheme 1006, a cytomorphologic model is trained and a histologic model is trained. Then, an ensemble model can be generated using the cytologic and histologic model together. For example, the cytomorphologic model and the histologic model may produce cytomorphologic confidence values and histologic confidence values correspondingly. The ensemble model may generate a final classification by aggregating these two confidence values. For example, for a model intended to be bias to high accuracy but low inclusivity (that is, one with few false positives at the cost of more false negatives) the ensemble model may return a classification if and only if both confidence values are above corresponding thresholds. For a model intended to be bias to high inclusivity but low accuracy (that is, one with few false negatives at the cost of more false positives) the ensemble model may return a classification if either confidence value is above a corresponding threshold, or if the sum of both confidence values is above a threshold value. As will be understood, different ensemble models may be useful for different purposes. A model with high inclusivity but low accuracy may be appropriate for a screening of the general population intended to direct patients to follow-up testing. On the other hand, a model with high accuracy but low inclusivity may be used as part of a chain of test conforming to Bayesian logic that are intended to identify particularly rare diagnoses of patients.
Therefore, in view of Allen, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the feature detection model comprises a plurality of models, each model detecting a certain class of feature, incorporated in the device of Conjeti, in order to utilize models to perform the detection of features and characterizing of the features, which using automated analyses of biologic specimens can decrease time and costs of analyses when compared to traditional methods (as stated in Allen ¶ [14]).
Re claim 14: However, Conjeti fails to specifically teach the features of the method of claim 12, wherein the model for characterising the features characterises all features in all classes for simultaneous consideration by the optimization model.
However, this is well known in the art as evidenced by Allen. Similar to the primary reference, Allen discloses a machine learning model that determines features in an image (same field of endeavor or reasonably pertinent to the problem).
Allen discloses wherein the model for characterising the features characterises all features in all classes for simultaneous consideration by the optimization model (e.g. separate models can be used to analyze features of different classes to be simultaneously considered by another model once received by both different models for different classes. This is taught in ¶ [99] above.).
Therefore, in view of Allen, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the model for characterising the features characterises all features in all classes for simultaneous consideration by the optimization model, incorporated in the device of Conjeti, in order to utilize models to perform the detection of features and characterizing of the features, which using automated analyses of biologic specimens can decrease time and costs of analyses when compared to traditional methods (as stated in Allen ¶ [14]).
Re claim 19: Conjeti discloses the device for analyzing medical scan images from a single patient of claim 17, wherein the optimisation model also uses a machine learning model (e.g. the abnormalities are passed to the determination unit with the images in order to determine changes of the abnormality. The determination unit uses a decision algorithm to perform this feature, which is taught in ¶ [60]-[62] and [69] above.).
However, Conjeti fails to specifically teach the features of the device of claim 17, wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model.
However, this is well known in the art as evidenced by Allen. Similar to the primary reference, Allen discloses a machine learning model that determines features in an image (same field of endeavor or reasonably pertinent to the problem).
Allen discloses wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model (e.g. the reference discloses using two machine learning models to perform different parts of the analysis. A first tier can be used to determine the features while a second tier is used to characterize what the scores associated with the features represent, which is taught in ¶ [47] and [55].).
Therefore, in view of Allen, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein detecting the features is done with a first machine learning model, and characterising the features is done with a second machine learning model, incorporated in the device of Conjeti, in order to utilize models to perform the detection of features and characterizing of the features, which using automated analyses of biologic specimens can decrease time and costs of analyses when compared to traditional methods (as stated in Allen ¶ [14]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Konen discloses setting a priority order of reading scans.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682
