DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-18 and 25-26 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-6, 14-18 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Neumann, etc. (US 20150242589 A1) in view of Hsieh, etc. (US 20190220975 A1), further in view of Amyot, etc. (US 20120128218 A1).
Regarding claim 1, Neumann teaches that a method implemented on at least one machine each of which has at least one processor and at least one storage device in a server of a medical image system for image display in a medical imaging system, wherein the medical image system includes an imaging device and a viewing device both connected to the server, and the method (See Neumann: Fig. 8, and [0054], “The above-described methods for estimating patient-specific parameters and corresponding uncertainty values of a computational model of organ function, patient-specific simulation of cardiac function, and generating a patient-specific computational model of the heart can be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components. A high-level block diagram of such a computer is illustrated in FIG. 8. Computer 802 contains a processor 804, which controls the overall operation of the computer 802 by executing computer program instructions which define such operation. The computer program instructions may be stored in a storage device 812 (e.g., magnetic disk) and loaded into memory 810 when execution of the computer program instructions is desired. Thus, the steps of the methods of FIGS. 1, 2, 4, and 5 may be defined by the computer program instructions stored in the memory 810 and/or storage 812 and controlled by the processor 804 executing the computer program instructions. An image acquisition device 820, such as an MR scanning device, Ultrasound device, etc., can be connected to the computer 802 to input image data to the computer 802. It is possible to implement the image acquisition device 820 and the computer 802 as one device. It is also possible that the image acquisition device 820 and the computer 802 communicate wirelessly through a network. The computer 802 also includes one or more network interfaces 806 for communicating with other devices via a network. The computer 802 also includes other input/output devices 808 that enable user interaction with the computer 802 (e.g., display, keyboard, mouse, speakers, buttons, etc.). Such input/output devices 808 may be used in conjunction with a set of computer programs as an annotation tool to annotate volumes received from the image acquisition device 820. One skilled in the art will recognize that an implementation of an actual computer could contain other components as well, and that FIG. 8 is a high level representation of some of the components of such a computer for illustrative purposes. For example, embodiments of the present invention may be implemented using multi-core architectures, multicomputer architecture, or a cloud based infrastructure, in addition to the computer system described in FIG. 8”) comprises:
constructing (See Neumann: Figs. 1-5, and [0020], “At step 110, a patient-specific computational heart model is generated based on the medical images and the clinical data of the patient. The patient-specific computational heart model is generated using an inverse problem to adjust parameters of the computational heart model such that simulated parameters, such as heart motion, ejection fraction, etc., output by the patient-specific computational heart model match the clinical data and medical images observed for the patient. According to an advantageous embodiment, uncertainty values are calculated for at least a subset of parameters of the computational heart model. For example, parameters of a model of cardiac function included in the computational heart model, such as a cardiac electrophysiology model, a cardiac hemodynamics model, or a cardiac biomechanics model, can estimated with uncertainty values”) a simulation model corresponding to the imaging device, the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device;
generating a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device, the module parameter adjustment instruction being configured to adjust one or more parameters of the simulation model (See Neumann: Figs. 1-5 and [0040], “At step 412, a most likely point in the set of GMMs is selected as the final estimate of parameter values. In particular, for each GMM, a support value is determined for the mean of each of the mixture components from the other GMMs. The means for all of the mixture models in all of the GMMs are then clustered based on their support values, a cluster with the highest combined support value is determined, and the centroid of the means in the cluster with the highest combine support value is selected. This centroid provides a point in the parameter space, which gives the final estimate of the values for the set of parameters for the computational model. That is, the center of the mode (cluster) corresponds to the estimated model parameters, and the local width of the mode its uncertainty”; [0046], “An exemplary implementation that was performed by the present inventors is described herein, but it is to be understood that the present invention is no limited to this specific example. The PCE-based stochastic surrogate model is estimated for the forward cardiac biomechanics model, as described above. In the exemplary implementation, to estimate the coefficients (.alpha..sub.p) for each response, 25 true model evaluations were performed on an isotropic grid of size 5.times.5 within the bounds .theta..sup.l=(400,100).sup.T and .theta..sup.u=(800,500).sup.T of the prior model distribution p(.theta.)=(.theta..sup.l,.theta..sup.u). The mean-shift based posterior analysis was carried out using 15 random noise levels, resulting in 15 GMMs. To estimate the GMMs based on the MCMC samples, the noise level for each response was drawn individually from a uniform distribution (r.sup.l,r.sup.u), where r.sup.l=5 and r.sup.u=25 are the heuristically chosen lower and upper bounds. The response noise levels are the error variances, which can be directly placed into the covariance matrix in Equation (3), which can be assumed to be diagonal. In the exemplary implementation, 50,000 MCMC sample were used for each GMM after discarding the first 10,000 samples as burn-in”; and [0018], “Embodiments of the present invention utilize a stochastic method to estimate patient-specific parameters for a computational heart model and their uncertainty due to noise in measurements used to personalize the model. While the forward model could be employed directly, should it be fast enough to compute, embodiments of the present invention utilize polynomial chaos expansion to estimate a surrogate for a model of cardiac function to make the problem computationally tractable. The surrogate model, or the forward model, is then applied with Bayesian inference to estimate posterior probabilities for different model parameter values. Embodiments of the present invention then utilize a strategy based on the mean-shift algorithm to find the optimal parameter values by exploring the space of measurement uncertainties”. Note that the detailed stochastic procedures for parameter estimating and adjustment is mapped to the simulation parameter adjustment instruction; and the noise measurement from the imaging device is used to estimate the model, and this is mapped to transmitting the imaging data to the model construction); and
outputting an adjusted simulation image based on an adjusted simulation model, the adjusted simulation image being used for displaying in the viewing device (See Neumann: Figs. 1-5 and 8, and [0042], “At step 416, the final parameter estimate and parameter regions are output. For example, the final parameter values and corresponding confidence regions can be output by displaying the parameter values and confidence regions on a display device of computer system”; [0046], “At step 510, patient-specific parameters of the cardiac biomechanics model and corresponding uncertainty values are estimated. The patient-specific parameters of the cardiac biomechanics model and the corresponding uncertainty values are estimated using the method of FIG. 4 described above. In an exemplary implementation, the parameters and uncertainty estimation can be performed for global passive and active biomechanical tissue properties, namely the Young modulus E for a linear elastic tissue model and the maximum myocyte contraction .sigma..sub.0, thus .theta.=(E,.sigma..sub.0). The responses d.sup.c can be features derived from time-series of left ventricular pressure and volume, in particular the minima, maxima, and means of the curves. Pressure data can be acquired through invasive catheterization and volume information can be obtained from cardiac cineMR by segmentation and tracking of the bi-ventricular myocardium. An exemplary implementation that was performed by the present inventors is described herein, but it is to be understood that the present invention is no limited to this specific example. The PCE-based stochastic surrogate model is estimated for the forward cardiac biomechanics model, as described above. In the exemplary implementation, to estimate the coefficients (.alpha..sub.p) for each response, 25 true model evaluations were performed on an isotropic grid of size 5.times.5 within the bounds .theta..sup.l=(400,100).sup.T and .theta..sup.u=(800,500).sup.T of the prior model distribution p(.theta.)=(.theta..sup.l,.theta..sup.u). The mean-shift based posterior analysis was carried out using 15 random noise levels, resulting in 15 GMMs. To estimate the GMMs based on the MCMC samples, the noise level for each response was drawn individually from a uniform distribution (r.sup.l,r.sup.u), where r.sup.l=5 and r.sup.u=25 are the heuristically chosen lower and upper bounds. The response noise levels are the error variances, which can be directly placed into the covariance matrix in Equation (3), which can be assumed to be diagonal. In the exemplary implementation, 50,000 MCMC sample were used for each GMM after discarding the first 10,000 samples as burn-in”; and [0050], “The computed parameters and their uncertainty are displayed to the user. Uncertainty can be displayed by a number, or through color coding (e.g. green for highly confident estimates, yellow, and red for low confidence). The forward model can also be run for various parameters varying within their confidence interval, thus propagation the uncertainty on the model output. If a therapy simulation is performed, the uncertainty can be propagated in a similar way to the therapy prediction, and a color code can be displayed to the user (along with the absolute uncertainty value)”. Note that the parameter estimation and displaying to the user is mapped to the outputting the simulation model results).
However, Neumann fails to explicitly disclose that a simulation model corresponding to the imaging device; the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device; and a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device.
However, Hsieh teaches that that a simulation model corresponding to the imaging device (See Hsieh: Figs. 14-15, and [0177], “Using the off-device processing engine(s) (e.g., the acquisition engine 1430, reconstruction engine 1440, diagnosis engine 1450, etc., and their associated deployed deep learning network devices 1522, 1532, and/or 1542, etc.), acquisition settings can be determined and sent to the imaging device 1410, for example. For example, purpose for exam, electronic medical record information, heart rate and/or heart rate variability, blood pressure, weight, visual assessment of prone/supine, head first or feet first, etc., can be used to determine one or more acquisition settings such as default field of view (DFOV), center, pitch, orientation, contrast injection rate, contrast injection timing, voltage, current, etc., thereby providing a “one-click” imaging device. Similarly, kernel information, slice thickness, slice interval, etc., can be used to determine one or more reconstruction parameters including image quality feedback, for example. Acquisition feedback, reconstruction feedback, etc., can be provided to the system design engine 1560 to provide real-time (or substantially real-time given processing and/or transmission delay) health analytics for the imaging device 1410 as represented by one or more digital models (e.g., deep learning models, machine models, digital twin, etc.). The digital model(s) can be used to predict component health for the imaging device 1410 in real-time (or substantially real time given a processing and/or transmission delay)”. Note that the digital twin is mapped to a simulation model corresponding to the imaging device); and
a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device (See Hsieh: Fig. 20, and [0215], “The deployed learning device 2050 is trained to respond to good (or sufficient) quality images and provide suggestions to the user 1404 when a poor (or insufficient) quality image is obtained. The device 2050 recognizes good image quality and suggests settings used to obtain a good quality image in a particular circumstance as default settings for that particular circumstance. When a bad quality image is obtained (e.g., through bad settings, user error, etc.), the device 2050 can suggest how to recover from the mistake, such as by suggesting different settings that can be changed to correct the mistake. Input parameters include default field of view (DFOV), center, voltage (kV), current (mA), pitch, orientation, injection rate, injection timing, etc. Rather than acquiring a scout image to identify landmarks and use those settings, deep learning through the deployed device 2050 can facilitate a one-click determination of scan range, field of view, and/or other settings, and the operator can modify or approve and activate the image acquisition, for example”. Note that the model is updated and the parameters, such as kV, mA, etc. are adjusted using data gathers from patient in real time is mapped to the current limitation of “a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Neumann to have a simulation model corresponding to the imaging device; and a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device as taught by Hsieh in order to improve diagnostic accuracy and coverage (See Hsieh: Fig. 1, and [0054], “Certain examples provide and/or facilitate improved imaging devices which improve diagnostic accuracy and/or coverage. Certain examples facilitate improved image acquisition and reconstruction to provide improved diagnostic accuracy. For example, image quality (IQ) metrics and automated validation can be facilitated using deep learning and/or other machine learning technologies”). Neumann teaches a method and system that may use the first AI model to reconstruct the human organ model based on the captured images and use a second AI model to generate the medical imaging configuration, and a third AI model to evaluate the second model, adjust the first model, and output the feedbacks to the respective first model and second model; while Hsieh teaches a system and method that may predict the market target of the games based on the analysis of the historical data of the game features. Therefore, it is obvious to one of ordinary skill in the art to modify Neumann by Hsieh to perform graph analysis on the historical data and get the predictor variable to predict the target variable. The motivation to modify Neumann by Hsieh is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Neumann, modified by Hsieh, fails to explicitly disclose that the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device.
However, Amyot teaches that the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device (See Amyot: Fig. 13, and [0056], “The processor 122 is adapted to generate a slice of the 3D model according to a cross-sectional plane 123 defined by the position and orientation of the probe 116, and to render an ultrasound image by using the slice. In one embodiment in which the 3D model stored in memory 120 comprises the surface model of the organ and the cloud of points to which a density value is associated, the processor 122 is adapted to render the ultrasound image using the cloud of points. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the points of the cloud. The resulting ultrasound image 124 is displayed on the monitor 114. Alternatively, the processor 122 is adapted to perform a voxelization of the slice, calculate a density value for each voxel, and render the ultrasound image using the voxelized model. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. In another embodiment in which the 3D model stored in memory 120 comprises voxels and associated data, the processor 122 is adapted to slice the 3D model and render an ultrasound image by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. The resulting ultrasound image 124 is displayed on the monitor 114”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Neumann to have the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device as taught by Amyot in order to ensure that surfaces and internal features of the organs can be recognized reliably for accurate diagnosis (See Amyot: Fig. 1, and [0006], “The method proposed herein involves the generation of a 3D model of an organ using volume modeling. Unlike wire frame and surface modeling, volume modeling systems ensure that all surfaces meet properly and that the object is geometrically correct. Volume modeling simulates an object internally and externally. Volumic 3D models can be sectioned to reveal their internal features. When an object is built as a 3D model, cross sections of its internal structure can be rendered as if it were sliced”). Neumann teaches a method and system that may use the first AI model to reconstruct the human organ model based on the captured images and use a second AI model to generate the medical imaging configuration, and a third AI model to evaluate the second model, adjust the first model, and output the feedbacks to the respective first model and second model; while Amyot teaches a system and method that may render the sliced images of the 3D model and display the rendered image to the user in real time. Therefore, it is obvious to one of ordinary skill in the art to modify Neumann by Amyot to generate and output the adjusted simulation images in sliced plane images for the user in real time for more accurate diagnostic evaluation of the patient’s organ functions. The motivation to modify Neumann by Amyot is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Hsieh teaches that the method of claim 1, wherein the constructing a simulation model corresponding to the imaging device comprises:
in response to receiving an initialization configuration instruction, constructing the simulation model corresponding to the imaging device (See Hsieh: Figs. 4A-B, and [0089], “In some examples, in operation, “weak” connections and nodes can initially be set to zero. The DLN 420 then processes its nodes in a retaining process. In certain examples, the nodes and connections that were set to zero are not allowed to change during the retraining. Given the redundancy present in the network 420, it is highly likely that equally good images will be generated. As illustrated in FIG. 4B, after retraining, the DLN 420 becomes DLN 421. DLN 421 is also examined to identify weak connections and nodes and set them to zero. This further retrained network is DLN 422. The example DLN 422 includes the “zeros” in DLN 421 and the new set of nodes and connections. The DLN 422 continues to repeat the processing until a good image quality is reached at a DLN 423, which is referred to as a “minimum viable net (MVN)”. The DLN 423 is a MVN because if additional connections or nodes are attempted to be set to zero in DLN 423, image quality can suffer”).
Regarding claim 3, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Hsieh teaches that the method of claim 1, wherein the simulation image further includes an identification content corresponding to the simulation image data (See Hsieh: Fig. 14, and [0120], “Using the example system 1400, the patient 1404 can be examined by the imaging system 1410 (e.g., CT, x-ray, MR, PET, ultrasound, MICT, single photon emission computed tomography (SPECT), digital tomosynthesis, etc.) based on settings from the information subsystem 1420 and/or acquisition engine 1430. Settings can be dictated and/or influenced by a deployed deep learning network model/device, such as CNN, RNN, etc. Based on information, such as a reason for exam, patient identification, patient context, population health information, etc., imaging device 1410 settings can be configured for image acquisition with respect to the patient 1406 by the acquisition engine 1430, alone or in conjunction with the information subsystem 1420 (e.g., a picture archiving and communication system (PACS), hospital information system (HIS), radiology information system (RIS), laboratory information system (LIS), cardiovascular information system (CVIS), etc.). The information from the information subsystem 1420 and/or acquisition engine 1430, as well as feedback from the imaging device 1410, can be collected and provided to a training deep learning network model to modify future settings, recommendations, etc., for image acquisition, for example. Periodically and/or upon satisfaction of certain criterion, the training deep learning network model can process the feedback and generate an updated model for deployment with respect to the system 1400.”), the identification content includes at least one of a device operation position, a device state parameter change content, a device operation description, or a next operation suggestion (See Hsieh: Fig. 19, and [0195], “At block 1904, the acquisition deployed deep learning network device 1522 analyzes the input to the acquisition engine 1430. For example, the DDLD 1522 processes patient parameters, prior imaging device 1410 scan parameters, etc., to generate imaging device 1410 settings for image acquisition. Using a CNN, RNN, autoencoder network, and/or other deep/machine learning network, the DLN 520 leverages prior acquisitions in comparison to current imaging device 1410 settings, patient information, reason for exam, patient history, and population health information, etc., to generate a predictive output. Relationships between settings, events, and results can be explored to determine appropriate imaging device 1410 settings, ideal or preferred acquisition settings based on type of exam and type of patient, changes to imaging device 1410 design, etc. Settings can include intensity or radiation dosage settings for sufficient (versus poor and/or versus high quality, etc.) image quality, etc. Settings can include acquisition type, duration, angle, number of scans, position, etc.”).
Regarding claim 5, Neumann, Hsieh, and Amyot teach all the features with respect to claim 2 as outlined above. Further, Amyot teaches that the method of claim 2, wherein the in response to receiving an initialization configuration instruction, constructing the simulation model corresponding to the imaging device comprises:
in response to the initialization configuration instruction sent by the imaging device, generating one or more model initialization configuration parameters, the one or more model initialization configuration parameters being configured for constructing the simulation model corresponding to the imaging device (See Amyot: Figs. 1A-B, and [0008], In accordance with a first broad aspect, there is provided a method for simulating an imaging process for an organ, the method comprising: retrieving from a memory a 3D volume model of the organ, the 3D volume model describing a 3D structure of the organ and a distribution of density within the 3D structure, the 3D structure representing a surface and internal features of the organ; generating a slice of the 3D model according to a position and an orientation of an imaging device, the slice including a cross-section of the surface and the internal features; rendering an image in accordance with the slice; and displaying the image [0028], “FIG. 1a illustrates one embodiment of a method 8 for simulating an ultrasound image of an organ. The first step 10 of the method 8 consists in the creation of a 3D volume model of the organ. The 3D model which may be stored in a memory comprises a 3D structure of the organ which includes both a 3D surface and the internal features of the organ. The 3D surface of the organ comprises the external surface of the organ and may also comprise internal surfaces. Data such as density is distributed within the 3D model. The internal features comprise elements contained within the external surface of the organ, such as muscle myocytes, veins, arteries, cavities, and the like. Taking the example of a heart, the 3D model takes into account internal features such as the endocardium, ventricular cavities, papillary muscles, valves, and the like. Furthermore, the 3D model may also be modified to include any pathology, even rare pathologie”; and [0029], “In one embodiment, the creation of the 3D model 10 comprises four steps, as illustrated in FIG. 1b. The first step 16 is the generation of a 3D structure of the organ. The 3D structure is an empty model of the organ representing the external outline of the organ. The empty model comprises the external surface of the organ. In other words, the first step 16 is the creation of the external envelop of the organ. The 3D structure may also comprise internal features of the organ. In this case, the empty model also comprises the surface of any internal features. Taking the example of a heart, FIG. 2 illustrates a cross-section 20 of an empty model of a heart. The empty model of the heart comprises the external surface 22 of the heart and the surface of internal cavities 24 and 26. It should be understood that the surface of the internal cavities 24 and 26 may be omitted in the 3D structure. Any surface modeling technique such as polygonal surface modeling or non-uniform rational B-spline modeling can be used to generate the empty model”. Note that retrieving and creating the various model is mapped to initialization of the model); and
generating the simulation image corresponding to the imaging device using the simulation model, the simulation image including the simulation image data of the imaging device, and the simulation image data being used for displaying in the viewing device (See Amyot: Figs. 1A-B, and [0044], “Referring back to FIG. 1a, step 12 of the method comprises the generation of a planar slice of the 3D model. The slice is generated according to a position and an orientation of a probe and has a given thickness. The thickness of the slice is chosen so that the simulated ultrasound image realistically reproduces a real ultrasound image. The position and orientation of the probe define a cross-sectional plane 52, as illustrated in FIG. 5. The slice is defined according to the cross-sectional plane. The next step 14 comprises the rendering of an ultrasound image using the slice generated at step 12. The volumes resulting from the voxelization and the density values associated with each voxel provide a virtual ultrasonographic texture to the ultrasound image. The resulting ultrasound image may be colored or greyscaled. The color or grey value of each pixel of the displayed image is defined by the density value associated with its corresponding voxel. If the thickness of the slice is large enough to comprise several voxels, each pixel of the displayed image is associated with several voxels. In this case, the density value and the color associated with this pixel is a function of the density values of all of the voxels to which this pixel is associated. For example, an average density value may be calculated and used to assign a color to the particular pixel of the displayed image. The last step 15 of the method 8 illustrated in FIG. 1a is the displaying of the rendered ultrasound image”).
Regarding claim 6, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Hsieh teaches that the method of claim 1, wherein the one or more device state parameters are outputted by the imaging device based on a digital twin technology (See Hsieh: Figs. 15A-B, and [0290], “At block 3308, system operation is monitored and modeled using the system design engine 1560. For example, gathered input data is formatted by the input formatter 1561 and processed using the model processor 1563 in conjunction with one or more models 1581-1587 from the model library 1580 associated with each component being modeled and monitored. For example, the data is used to form and/or modify nodes and/or connections between nodes in a deep learning network, such as a deep convolutional neural network, auto-encoder network, deep residual network, machine learning network, etc., embodied in one or more models 1581-1587 in the library or catalog 1580 (as described above). Weights and/or biases associated with nodes, connections, etc., can also be modified by patterns, relationships, values, presence or absence of values, etc., found by the model(s) 1581-1587 in the input data, for example. Each model 1581-1587, taken alone or in combination (e.g., connected as the corresponding system components are connected in the target system 1500 to form a digital model, digital twin, etc.), can be used by the model processor 1563 to simulate component(s) and/or overall system 1500 operation, given the received input”).
Regarding claim 8, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Hsieh teaches that the method of claim 1, further comprising:
in response to receiving an interactive instruction, outputting simulation image adjustment information, the simulation image adjustment information being configured for instructing the server to adjust the simulation model (See Hsieh: Figs. 8A-D, and [0100], “The deployed device 703 operates on input and provides output, and a feedback collector 808 monitors the output (and input) and gathers feedback based on operation of the deployed deep learning device 703. The feedback is stored in feedback storage 810 until a certain amount of feedback has been collected (e.g., a certain quantity, a certain quality/consistency, a certain time period, etc.). Once sufficient feedback has been collected, a re-training initiator 812 is triggered. The re-training initiator 812 retrieves data from the feedback storage 810 and operates in conjunction with a re-training data selector 814 to select data from the feedback storage 810 to provide to the training deep learning device 701. The network of the training device 701 is then updated/re-trained using the feedback until the model evaluator 802 is satisfied that the training network model is complete. The updated/re-trained model is then prepared and deployed in the deployed deep learning device 703 as described above”; and Fig. 14, and [0120], “Using the example system 1400, the patient 1404 can be examined by the imaging system 1410 (e.g., CT, x-ray, MR, PET, ultrasound, MICT, single photon emission computed tomography (SPECT), digital tomosynthesis, etc.) based on settings from the information subsystem 1420 and/or acquisition engine 1430. Settings can be dictated and/or influenced by a deployed deep learning network model/device, such as CNN, RNN, etc. Based on information, such as a reason for exam, patient identification, patient context, population health information, etc., imaging device 1410 settings can be configured for image acquisition with respect to the patient 1406 by the acquisition engine 1430, alone or in conjunction with the information subsystem 1420 (e.g., a picture archiving and communication system (PACS), hospital information system (HIS), radiology information system (RIS), laboratory information system (LIS), cardiovascular information system (CVIS), etc.). The information from the information subsystem 1420 and/or acquisition engine 1430, as well as feedback from the imaging device 1410, can be collected and provided to a training deep learning network model to modify future settings, recommendations, etc., for image acquisition, for example. Periodically and/or upon satisfaction of certain criterion, the training deep learning network model can process the feedback and generate an updated model for deployment with respect to the system 1400”. Note that based on the feedback, and the recommendation, retraining and deploying the simulation model is mapped to the interactive instructions).
Regarding claim 9, Neumann, Hsieh, and Amyot teach all the features with respect to claim 8 as outlined above. Further, Neumann teaches that the method of claim 8, further comprising:
outputting one or more device state adjustment parameters based on the simulation image adjustment information, the one or more device state adjustment parameters being configured for instructing the imaging device to make a corresponding operational state adjustment (See Hsieh: Figs. 8A-D, and [0100], “The deployed device 703 operates on input and provides output, and a feedback collector 808 monitors the output (and input) and gathers feedback based on operation of the deployed deep learning device 703. The feedback is stored in feedback storage 810 until a certain amount of feedback has been collected (e.g., a certain quantity, a certain quality/consistency, a certain time period, etc.). Once sufficient feedback has been collected, a re-training initiator 812 is triggered. The re-training initiator 812 retrieves data from the feedback storage 810 and operates in conjunction with a re-training data selector 814 to select data from the feedback storage 810 to provide to the training deep learning device 701. The network of the training device 701 is then updated/re-trained using the feedback until the model evaluator 802 is satisfied that the training network model is complete. The updated/re-trained model is then prepared and deployed in the deployed deep learning device 703 as described above”; and [0103], “At block 832, feedback from operation of the deployed deep learning model-based device is collected and stored until the collected feedback satisfies a threshold (block 834). Feedback can include input, deployed model information, pre- and/or post-processing information, actual and/or corrected output, etc. Once the feedback collection threshold is satisfied, at block 836, model re-training is initiated. At block 838, data from the collected feedback (and/or other input data) is selected to re-train the deep learning model. Data selection can include pre- and/or post-processing to properly format the data for model training, etc. Control then passes to block 822 to (re)train the deep learning network model”. Note that the feedback is stored, and used to trigger the model retraining, and this is mapped to the outputting the device stat adjustment parameters).
Regarding claim 14, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Amyot teaches that the method of claim 1, wherein the imaging device is a handheld ultrasound device (See Amyot: Figs. 1A-B, and [0044], “Referring back to FIG. 1a, step 12 of the method comprises the generation of a planar slice of the 3D model. The slice is generated according to a position and an orientation of a probe and has a given thickness. The thickness of the slice is chosen so that the simulated ultrasound image realistically reproduces a real ultrasound image. The position and orientation of the probe define a cross-sectional plane 52, as illustrated in FIG. 5. The slice is defined according to the cross-sectional plane. The next step 14 comprises the rendering of an ultrasound image using the slice generated at step 12. The volumes resulting from the voxelization and the density values associated with each voxel provide a virtual ultrasonographic texture to the ultrasound image. The resulting ultrasound image may be colored or greyscaled. The color or grey value of each pixel of the displayed image is defined by the density value associated with its corresponding voxel. If the thickness of the slice is large enough to comprise several voxels, each pixel of the displayed image is associated with several voxels. In this case, the density value and the color associated with this pixel is a function of the density values of all of the voxels to which this pixel is associated. For example, an average density value may be calculated and used to assign a color to the particular pixel of the displayed image. The last step 15 of the method 8 illustrated in FIG. 1a is the displaying of the rendered ultrasound image”).
Regarding claim 15, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Neumann, Hsieh, and Amyot teach that a method implemented on at least one machine each of which has at least one processor and at least one storage device in a server of a medical image system for image display in a medical imaging system, wherein the medical image system includes an imaging device and a viewing device both connected to the server, and the method (See Neumann: Fig. 8, and [0054], “The above-described methods for estimating patient-specific parameters and corresponding uncertainty values of a computational model of organ function, patient-specific simulation of cardiac function, and generating a patient-specific computational model of the heart can be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components. A high-level block diagram of such a computer is illustrated in FIG. 8. Computer 802 contains a processor 804, which controls the overall operation of the computer 802 by executing computer program instructions which define such operation. The computer program instructions may be stored in a storage device 812 (e.g., magnetic disk) and loaded into memory 810 when execution of the computer program instructions is desired. Thus, the steps of the methods of FIGS. 1, 2, 4, and 5 may be defined by the computer program instructions stored in the memory 810 and/or storage 812 and controlled by the processor 804 executing the computer program instructions. An image acquisition device 820, such as an MR scanning device, Ultrasound device, etc., can be connected to the computer 802 to input image data to the computer 802. It is possible to implement the image acquisition device 820 and the computer 802 as one device. It is also possible that the image acquisition device 820 and the computer 802 communicate wirelessly through a network. The computer 802 also includes one or more network interfaces 806 for communicating with other devices via a network. The computer 802 also includes other input/output devices 808 that enable user interaction with the computer 802 (e.g., display, keyboard, mouse, speakers, buttons, etc.). Such input/output devices 808 may be used in conjunction with a set of computer programs as an annotation tool to annotate volumes received from the image acquisition device 820. One skilled in the art will recognize that an implementation of an actual computer could contain other components as well, and that FIG. 8 is a high level representation of some of the components of such a computer for illustrative purposes. For example, embodiments of the present invention may be implemented using multi-core architectures, multicomputer architecture, or a cloud based infrastructure, in addition to the computer system described in FIG. 8”) comprises:
constructing (See Neumann: Figs. 1-5, and [0020], “At step 110, a patient-specific computational heart model is generated based on the medical images and the clinical data of the patient. The patient-specific computational heart model is generated using an inverse problem to adjust parameters of the computational heart model such that simulated parameters, such as heart motion, ejection fraction, etc., output by the patient-specific computational heart model match the clinical data and medical images observed for the patient. According to an advantageous embodiment, uncertainty values are calculated for at least a subset of parameters of the computational heart model. For example, parameters of a model of cardiac function included in the computational heart model, such as a cardiac electrophysiology model, a cardiac hemodynamics model, or a cardiac biomechanics model, can estimated with uncertainty values”) a simulation model corresponding to the imaging device (See Hsieh: Figs. 14-15, and [0177], “Using the off-device processing engine(s) (e.g., the acquisition engine 1430, reconstruction engine 1440, diagnosis engine 1450, etc., and their associated deployed deep learning network devices 1522, 1532, and/or 1542, etc.), acquisition settings can be determined and sent to the imaging device 1410, for example. For example, purpose for exam, electronic medical record information, heart rate and/or heart rate variability, blood pressure, weight, visual assessment of prone/supine, head first or feet first, etc., can be used to determine one or more acquisition settings such as default field of view (DFOV), center, pitch, orientation, contrast injection rate, contrast injection timing, voltage, current, etc., thereby providing a “one-click” imaging device. Similarly, kernel information, slice thickness, slice interval, etc., can be used to determine one or more reconstruction parameters including image quality feedback, for example. Acquisition feedback, reconstruction feedback, etc., can be provided to the system design engine 1560 to provide real-time (or substantially real-time given processing and/or transmission delay) health analytics for the imaging device 1410 as represented by one or more digital models (e.g., deep learning models, machine models, digital twin, etc.). The digital model(s) can be used to predict component health for the imaging device 1410 in real-time (or substantially real time given a processing and/or transmission delay)”. Note that the digital twin is mapped to a simulation model corresponding to the imaging device);
generating a simulation image corresponding to the imaging device based on the simulation model, wherein the simulation image includes simulation image data of the imaging device (See Amyot: Fig. 13, and [0056], “The processor 122 is adapted to generate a slice of the 3D model according to a cross-sectional plane 123 defined by the position and orientation of the probe 116, and to render an ultrasound image by using the slice. In one embodiment in which the 3D model stored in memory 120 comprises the surface model of the organ and the cloud of points to which a density value is associated, the processor 122 is adapted to render the ultrasound image using the cloud of points. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the points of the cloud. The resulting ultrasound image 124 is displayed on the monitor 114. Alternatively, the processor 122 is adapted to perform a voxelization of the slice, calculate a density value for each voxel, and render the ultrasound image using the voxelized model. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. In another embodiment in which the 3D model stored in memory 120 comprises voxels and associated data, the processor 122 is adapted to slice the 3D model and render an ultrasound image by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. The resulting ultrasound image 124 is displayed on the monitor 114”);
obtaining, in real time, one or more device state parameters of the imaging device (See Hsieh: Fig. 20, and [0215], “The deployed learning device 2050 is trained to respond to good (or sufficient) quality images and provide suggestions to the user 1404 when a poor (or insufficient) quality image is obtained. The device 2050 recognizes good image quality and suggests settings used to obtain a good quality image in a particular circumstance as default settings for that particular circumstance. When a bad quality image is obtained (e.g., through bad settings, user error, etc.), the device 2050 can suggest how to recover from the mistake, such as by suggesting different settings that can be changed to correct the mistake. Input parameters include default field of view (DFOV), center, voltage (kV), current (mA), pitch, orientation, injection rate, injection timing, etc. Rather than acquiring a scout image to identify landmarks and use those settings, deep learning through the deployed device 2050 can facilitate a one-click determination of scan range, field of view, and/or other settings, and the operator can modify or approve and activate the image acquisition, for example”. Note that the model is updated and the parameters, such as kV, mA, etc. are adjusted using data gathers from patient in real time is mapped to the current limitation of “a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device”); and
updating the simulation image in real time based on the one or more device state parameters (See Neumann: Figs. 1-5 and 8, and [0042], “At step 416, the final parameter estimate and parameter regions are output. For example, the final parameter values and corresponding confidence regions can be output by displaying the parameter values and confidence regions on a display device of computer system”; [0046], “At step 510, patient-specific parameters of the cardiac biomechanics model and corresponding uncertainty values are estimated. The patient-specific parameters of the cardiac biomechanics model and the corresponding uncertainty values are estimated using the method of FIG. 4 described above. In an exemplary implementation, the parameters and uncertainty estimation can be performed for global passive and active biomechanical tissue properties, namely the Young modulus E for a linear elastic tissue model and the maximum myocyte contraction .sigma..sub.0, thus .theta.=(E,.sigma..sub.0). The responses d.sup.c can be features derived from time-series of left ventricular pressure and volume, in particular the minima, maxima, and means of the curves. Pressure data can be acquired through invasive catheterization and volume information can be obtained from cardiac cineMR by segmentation and tracking of the bi-ventricular myocardium. An exemplary implementation that was performed by the present inventors is described herein, but it is to be understood that the present invention is no limited to this specific example. The PCE-based stochastic surrogate model is estimated for the forward cardiac biomechanics model, as described above. In the exemplary implementation, to estimate the coefficients (.alpha..sub.p) for each response, 25 true model evaluations were performed on an isotropic grid of size 5.times.5 within the bounds .theta..sup.l=(400,100).sup.T and .theta..sup.u=(800,500).sup.T of the prior model distribution p(.theta.)=(.theta..sup.l,.theta..sup.u). The mean-shift based posterior analysis was carried out using 15 random noise levels, resulting in 15 GMMs. To estimate the GMMs based on the MCMC samples, the noise level for each response was drawn individually from a uniform distribution (r.sup.l,r.sup.u), where r.sup.l=5 and r.sup.u=25 are the heuristically chosen lower and upper bounds. The response noise levels are the error variances, which can be directly placed into the covariance matrix in Equation (3), which can be assumed to be diagonal. In the exemplary implementation, 50,000 MCMC sample were used for each GMM after discarding the first 10,000 samples as burn-in”; and [0050], “The computed parameters and their uncertainty are displayed to the user. Uncertainty can be displayed by a number, or through color coding (e.g. green for highly confident estimates, yellow, and red for low confidence). The forward model can also be run for various parameters varying within their confidence interval, thus propagation the uncertainty on the model output. If a therapy simulation is performed, the uncertainty can be propagated in a similar way to the therapy prediction, and a color code can be displayed to the user (along with the absolute uncertainty value)”. Note that the parameter estimation and displaying to the user is mapped to the outputting the simulation model results).
Regarding claim 16, Neumann, Hsieh, and Amyot teach all the features with respect to claim 15 as outlined above. Further, Hsieh teaches that the method of claim 15, wherein the constructing a simulation model corresponding to the imaging device comprises:
in response to receiving an initialization configuration instruction, constructing the simulation model corresponding to the imaging device (See Hsieh: Figs. 4A-B, and [0089], “In some examples, in operation, “weak” connections and nodes can initially be set to zero. The DLN 420 then processes its nodes in a retaining process. In certain examples, the nodes and connections that were set to zero are not allowed to change during the retraining. Given the redundancy present in the network 420, it is highly likely that equally good images will be generated. As illustrated in FIG. 4B, after retraining, the DLN 420 becomes DLN 421. DLN 421 is also examined to identify weak connections and nodes and set them to zero. This further retrained network is DLN 422. The example DLN 422 includes the “zeros” in DLN 421 and the new set of nodes and connections. The DLN 422 continues to repeat the processing until a good image quality is reached at a DLN 423, which is referred to as a “minimum viable net (MVN)”. The DLN 423 is a MVN because if additional connections or nodes are attempted to be set to zero in DLN 423, image quality can suffer”).
Regarding claim 17, Neumann, Hsieh, and Amyot teach all the features with respect to claim 15 as outlined above. Further, Hsieh teaches that the method of claim 15, wherein the simulation image further includes identification content corresponding to the simulation image data (See Hsieh: Fig. 14, and [0120], “Using the example system 1400, the patient 1404 can be examined by the imaging system 1410 (e.g., CT, x-ray, MR, PET, ultrasound, MICT, single photon emission computed tomography (SPECT), digital tomosynthesis, etc.) based on settings from the information subsystem 1420 and/or acquisition engine 1430. Settings can be dictated and/or influenced by a deployed deep learning network model/device, such as CNN, RNN, etc. Based on information, such as a reason for exam, patient identification, patient context, population health information, etc., imaging device 1410 settings can be configured for image acquisition with respect to the patient 1406 by the acquisition engine 1430, alone or in conjunction with the information subsystem 1420 (e.g., a picture archiving and communication system (PACS), hospital information system (HIS), radiology information system (RIS), laboratory information system (LIS), cardiovascular information system (CVIS), etc.). The information from the information subsystem 1420 and/or acquisition engine 1430, as well as feedback from the imaging device 1410, can be collected and provided to a training deep learning network model to modify future settings, recommendations, etc., for image acquisition, for example. Periodically and/or upon satisfaction of certain criterion, the training deep learning network model can process the feedback and generate an updated model for deployment with respect to the system 1400.”), and the identification content includes at least one of a device operation position, a device state parameter change content, a device operation description, or a next operation suggestion (See Hsieh: Fig. 19, and [0195], “At block 1904, the acquisition deployed deep learning network device 1522 analyzes the input to the acquisition engine 1430. For example, the DDLD 1522 processes patient parameters, prior imaging device 1410 scan parameters, etc., to generate imaging device 1410 settings for image acquisition. Using a CNN, RNN, autoencoder network, and/or other deep/machine learning network, the DLN 520 leverages prior acquisitions in comparison to current imaging device 1410 settings, patient information, reason for exam, patient history, and population health information, etc., to generate a predictive output. Relationships between settings, events, and results can be explored to determine appropriate imaging device 1410 settings, ideal or preferred acquisition settings based on type of exam and type of patient, changes to imaging device 1410 design, etc. Settings can include intensity or radiation dosage settings for sufficient (versus poor and/or versus high quality, etc.) image quality, etc. Settings can include acquisition type, duration, angle, number of scans, position, etc.”).
Regarding claim 18, Neumann, Hsieh, and Amyot teach all the features with respect to claim 15 as outlined above. Further, Hsieh teaches that the method of claim 15, wherein the updating the simulation image in real time based on the one or more device state parameters comprises:
generating, in real time, a module parameter adjustment instruction based on the one or more device state parameters, the module parameter adjustment instruction being used for adjusting one or more parameters of the simulation model (See Hsieh: Figs. 15A-B, and [0159], “Thus, the example system 1500 creates one or more images using interconnected DDLDs 1522, 1532, 1542 and corresponding engines 1430, 1440, and links the image(s) to decision support via the diagnosis engine 1450 and DDLD 1542 for diagnosis. Real-time (or substantially real time given processing and transmission delay) feedback (e.g., feed forward and feed back between learning and improvement factories 1520, 1530, 1540 and engines 1430, 1440, 1450) loops are formed in the example system 1500 between acquisition and reconstruction and between diagnosis and reconstruction, for example, for ongoing improvement of settings and operation of the acquisition engine 1430, reconstruction engine 1440, and diagnosis engine 1450 (e.g., directly and/or by replacing/updating the DDLD 1522, 1532, 1542 based on an updated/retrained DLN, etc.). As the system 1500 learns from the operation of its components, the system 1500 can improve its function. The user 1404 can also provide offline feedback (e.g., to the factory 1520, 1530, 1540, etc.). As a result, each factory 1520, 1530, 1540 learns differently based on system 1500 input as well as user input in conjunction with personalized variables associated with the patient 1406, for example”); and
updating, in real time, the simulation image based on an adjusted simulation model (See VVV: Figs. 14-15, and [0163], “FIG. 15B illustrates an example system implementation 1501 in which the acquisition engine 1430, reconstruction engine 1440, and diagnosis engine 1450 are accompanied by a data quality assessment engine 1570, an image quality assessment engine 1572, and a diagnosis assessment engine 1574. In the configuration 1501 of FIG. 15B, each engine 1430, 1440, 1450 receives direct feedback from an associated assessment engine 1570, 1572, 1574. In certain examples, the acquisition engine 1430, reconstruction engine 1440, and/or diagnosis engine 1450 receives feedback without having to update their associated deployed deep learning modules 1522, 1532, 1542. Alternatively or in addition, the data quality assessment engine 1570, image quality assessment engine 1572, and diagnosis assessment engine 1574 provide feedback to the engines 1430-1450. Although direct connections are depicted in the example of FIG. 15B for the sake of simplicity, it should be understood that each of the deep learning-based feedback modules 1570-1574 has an associated training image database including different classes of example conditions, an associated learning and improvement factory module, an orchestration module, and a trigger for associated parameter update and restart, for example”).
Regarding claim 25, Neumann, Hsieh, and Amyot teach all the features with respect to claim 15 as outlined above. Further, Neumann, Hsieh, and Amyot teach that the non-transitory computer-readable storage medium storing computer instructions, wherein when reading the computer instructions in the storage medium, a computer implements the method for image display in a medical imaging system, wherein the medical image system includes an imaging device and a viewing device both connected to the server, and the method (See Neumann: Fig. 8, and [0054], “The above-described methods for estimating patient-specific parameters and corresponding uncertainty values of a computational model of organ function, patient-specific simulation of cardiac function, and generating a patient-specific computational model of the heart can be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components. A high-level block diagram of such a computer is illustrated in FIG. 8. Computer 802 contains a processor 804, which controls the overall operation of the computer 802 by executing computer program instructions which define such operation. The computer program instructions may be stored in a storage device 812 (e.g., magnetic disk) and loaded into memory 810 when execution of the computer program instructions is desired. Thus, the steps of the methods of FIGS. 1, 2, 4, and 5 may be defined by the computer program instructions stored in the memory 810 and/or storage 812 and controlled by the processor 804 executing the computer program instructions. An image acquisition device 820, such as an MR scanning device, Ultrasound device, etc., can be connected to the computer 802 to input image data to the computer 802. It is possible to implement the image acquisition device 820 and the computer 802 as one device. It is also possible that the image acquisition device 820 and the computer 802 communicate wirelessly through a network. The computer 802 also includes one or more network interfaces 806 for communicating with other devices via a network. The computer 802 also includes other input/output devices 808 that enable user interaction with the computer 802 (e.g., display, keyboard, mouse, speakers, buttons, etc.). Such input/output devices 808 may be used in conjunction with a set of computer programs as an annotation tool to annotate volumes received from the image acquisition device 820. One skilled in the art will recognize that an implementation of an actual computer could contain other components as well, and that FIG. 8 is a high level representation of some of the components of such a computer for illustrative purposes. For example, embodiments of the present invention may be implemented using multi-core architectures, multicomputer architecture, or a cloud based infrastructure, in addition to the computer system described in FIG. 8”) comprises:
constructing (See Neumann: Figs. 1-5, and [0020], “At step 110, a patient-specific computational heart model is generated based on the medical images and the clinical data of the patient. The patient-specific computational heart model is generated using an inverse problem to adjust parameters of the computational heart model such that simulated parameters, such as heart motion, ejection fraction, etc., output by the patient-specific computational heart model match the clinical data and medical images observed for the patient. According to an advantageous embodiment, uncertainty values are calculated for at least a subset of parameters of the computational heart model. For example, parameters of a model of cardiac function included in the computational heart model, such as a cardiac electrophysiology model, a cardiac hemodynamics model, or a cardiac biomechanics model, can estimated with uncertainty values”) a simulation model corresponding to the imaging device (See Hsieh: Figs. 14-15, and [0177], “Using the off-device processing engine(s) (e.g., the acquisition engine 1430, reconstruction engine 1440, diagnosis engine 1450, etc., and their associated deployed deep learning network devices 1522, 1532, and/or 1542, etc.), acquisition settings can be determined and sent to the imaging device 1410, for example. For example, purpose for exam, electronic medical record information, heart rate and/or heart rate variability, blood pressure, weight, visual assessment of prone/supine, head first or feet first, etc., can be used to determine one or more acquisition settings such as default field of view (DFOV), center, pitch, orientation, contrast injection rate, contrast injection timing, voltage, current, etc., thereby providing a “one-click” imaging device. Similarly, kernel information, slice thickness, slice interval, etc., can be used to determine one or more reconstruction parameters including image quality feedback, for example. Acquisition feedback, reconstruction feedback, etc., can be provided to the system design engine 1560 to provide real-time (or substantially real-time given processing and/or transmission delay) health analytics for the imaging device 1410 as represented by one or more digital models (e.g., deep learning models, machine models, digital twin, etc.). The digital model(s) can be used to predict component health for the imaging device 1410 in real-time (or substantially real time given a processing and/or transmission delay)”. Note that the digital twin is mapped to a simulation model corresponding to the imaging device), the simulation model being configured to generate a simulation image corresponding to the imaging device, and the simulation image including simulation image data of the imaging device (See Amyot: Fig. 13, and [0056], “The processor 122 is adapted to generate a slice of the 3D model according to a cross-sectional plane 123 defined by the position and orientation of the probe 116, and to render an ultrasound image by using the slice. In one embodiment in which the 3D model stored in memory 120 comprises the surface model of the organ and the cloud of points to which a density value is associated, the processor 122 is adapted to render the ultrasound image using the cloud of points. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the points of the cloud. The resulting ultrasound image 124 is displayed on the monitor 114. Alternatively, the processor 122 is adapted to perform a voxelization of the slice, calculate a density value for each voxel, and render the ultrasound image using the voxelized model. The ultrasound image is generated by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. In another embodiment in which the 3D model stored in memory 120 comprises voxels and associated data, the processor 122 is adapted to slice the 3D model and render an ultrasound image by assigning a color or a grey scale value to each point of the slice in accordance with the density values of the voxels associated with the particular point. The resulting ultrasound image 124 is displayed on the monitor 114”);
generating a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device (See Hsieh: Fig. 20, and [0215], “The deployed learning device 2050 is trained to respond to good (or sufficient) quality images and provide suggestions to the user 1404 when a poor (or insufficient) quality image is obtained. The device 2050 recognizes good image quality and suggests settings used to obtain a good quality image in a particular circumstance as default settings for that particular circumstance. When a bad quality image is obtained (e.g., through bad settings, user error, etc.), the device 2050 can suggest how to recover from the mistake, such as by suggesting different settings that can be changed to correct the mistake. Input parameters include default field of view (DFOV), center, voltage (kV), current (mA), pitch, orientation, injection rate, injection timing, etc. Rather than acquiring a scout image to identify landmarks and use those settings, deep learning through the deployed device 2050 can facilitate a one-click determination of scan range, field of view, and/or other settings, and the operator can modify or approve and activate the image acquisition, for example”. Note that the model is updated and the parameters, such as kV, mA, etc. are adjusted using data gathers from patient in real time is mapped to the current limitation of “a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device”), the module parameter adjustment instruction being configured to adjust one or more parameters of the simulation model (See Neumann: Figs. 1-5 and [0040], “At step 412, a most likely point in the set of GMMs is selected as the final estimate of parameter values. In particular, for each GMM, a support value is determined for the mean of each of the mixture components from the other GMMs. The means for all of the mixture models in all of the GMMs are then clustered based on their support values, a cluster with the highest combined support value is determined, and the centroid of the means in the cluster with the highest combine support value is selected. This centroid provides a point in the parameter space, which gives the final estimate of the values for the set of parameters for the computational model. That is, the center of the mode (cluster) corresponds to the estimated model parameters, and the local width of the mode its uncertainty”; [0046], “An exemplary implementation that was performed by the present inventors is described herein, but it is to be understood that the present invention is no limited to this specific example. The PCE-based stochastic surrogate model is estimated for the forward cardiac biomechanics model, as described above. In the exemplary implementation, to estimate the coefficients (.alpha..sub.p) for each response, 25 true model evaluations were performed on an isotropic grid of size 5.times.5 within the bounds .theta..sup.l=(400,100).sup.T and .theta..sup.u=(800,500).sup.T of the prior model distribution p(.theta.)=(.theta..sup.l,.theta..sup.u). The mean-shift based posterior analysis was carried out using 15 random noise levels, resulting in 15 GMMs. To estimate the GMMs based on the MCMC samples, the noise level for each response was drawn individually from a uniform distribution (r.sup.l,r.sup.u), where r.sup.l=5 and r.sup.u=25 are the heuristically chosen lower and upper bounds. The response noise levels are the error variances, which can be directly placed into the covariance matrix in Equation (3), which can be assumed to be diagonal. In the exemplary implementation, 50,000 MCMC sample were used for each GMM after discarding the first 10,000 samples as burn-in”; and [0018], “Embodiments of the present invention utilize a stochastic method to estimate patient-specific parameters for a computational heart model and their uncertainty due to noise in measurements used to personalize the model. While the forward model could be employed directly, should it be fast enough to compute, embodiments of the present invention utilize polynomial chaos expansion to estimate a surrogate for a model of cardiac function to make the problem computationally tractable. The surrogate model, or the forward model, is then applied with Bayesian inference to estimate posterior probabilities for different model parameter values. Embodiments of the present invention then utilize a strategy based on the mean-shift algorithm to find the optimal parameter values by exploring the space of measurement uncertainties”. Note that the detailed stochastic procedures for parameter estimating and adjustment is mapped to the simulation parameter adjustment instruction; and the noise measurement from the imaging device is used to estimate the model, and this is mapped to transmitting the imaging data to the model construction); and
outputting an adjusted simulation image based on an adjusted simulation model, the adjusted simulation image being used for displaying in the viewing device (See Neumann: Figs. 1-5 and 8, and [0042], “At step 416, the final parameter estimate and parameter regions are output. For example, the final parameter values and corresponding confidence regions can be output by displaying the parameter values and confidence regions on a display device of computer system”; [0046], “At step 510, patient-specific parameters of the cardiac biomechanics model and corresponding uncertainty values are estimated. The patient-specific parameters of the cardiac biomechanics model and the corresponding uncertainty values are estimated using the method of FIG. 4 described above. In an exemplary implementation, the parameters and uncertainty estimation can be performed for global passive and active biomechanical tissue properties, namely the Young modulus E for a linear elastic tissue model and the maximum myocyte contraction .sigma..sub.0, thus .theta.=(E,.sigma..sub.0). The responses d.sup.c can be features derived from time-series of left ventricular pressure and volume, in particular the minima, maxima, and means of the curves. Pressure data can be acquired through invasive catheterization and volume information can be obtained from cardiac cineMR by segmentation and tracking of the bi-ventricular myocardium. An exemplary implementation that was performed by the present inventors is described herein, but it is to be understood that the present invention is no limited to this specific example. The PCE-based stochastic surrogate model is estimated for the forward cardiac biomechanics model, as described above. In the exemplary implementation, to estimate the coefficients (.alpha..sub.p) for each response, 25 true model evaluations were performed on an isotropic grid of size 5.times.5 within the bounds .theta..sup.l=(400,100).sup.T and .theta..sup.u=(800,500).sup.T of the prior model distribution p(.theta.)=(.theta..sup.l,.theta..sup.u). The mean-shift based posterior analysis was carried out using 15 random noise levels, resulting in 15 GMMs. To estimate the GMMs based on the MCMC samples, the noise level for each response was drawn individually from a uniform distribution (r.sup.l,r.sup.u), where r.sup.l=5 and r.sup.u=25 are the heuristically chosen lower and upper bounds. The response noise levels are the error variances, which can be directly placed into the covariance matrix in Equation (3), which can be assumed to be diagonal. In the exemplary implementation, 50,000 MCMC sample were used for each GMM after discarding the first 10,000 samples as burn-in”; and [0050], “The computed parameters and their uncertainty are displayed to the user. Uncertainty can be displayed by a number, or through color coding (e.g. green for highly confident estimates, yellow, and red for low confidence). The forward model can also be run for various parameters varying within their confidence interval, thus propagation the uncertainty on the model output. If a therapy simulation is performed, the uncertainty can be propagated in a similar way to the therapy prediction, and a color code can be displayed to the user (along with the absolute uncertainty value)”. Note that the parameter estimation and displaying to the user is mapped to the outputting the simulation model results).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Neumann, etc. (US 20150242589 A1) in view of Hsieh, etc. (US 20190220975 A1), further in view of Amyot, etc. (US 20120128218 A1) and Imai, etc. (US 20200033461 A1).
Regarding claim 7, Neumann, Hsieh, and Amyot teach all the features with respect to claim 2 as outlined above. However, Neumann, modified by Hsieh and Amyot fails to explicitly disclose that the method of claim 2, wherein the initialization configuration instruction is determined by: determining the initialization configuration instruction in a situation where the imaging device is detected to be switched from a non-operational state to an operational state, the initialization configuration instruction being configured for instructing the server to construct a simulation model corresponding to the imaging device.
However, Imai teaches that the method of claim 2, wherein the initialization configuration instruction is determined by: determining the initialization configuration instruction in a situation where the imaging device is detected to be switched from a non-operational state to an operational state, the initialization configuration instruction being configured for instructing the server to construct a simulation model corresponding to the imaging device (See Imai: Fig. 1, and [0022], “In one embodiment, the processor 102 in the imaging unit and the microcontroller 62 in the transducer are programmed to detect the pre-programmed, pre-defined or user-defined operating conditions that cause a transition between power states. In one embodiment, transitioning from the normal operating state 200 where the imaging system is actively obtaining and displaying ultrasound data from a region of interest to a low frame rate state 204 that captures ultrasound signals at a lower rate occurs if one or more conditions 250 are met such as the transducer probe temperature being above a set limit (e.g. >41 C) or if the transducer probe has been in contact with a patient's skin for more than a defined limit (e.g. more than 3 minutes without the transducer moving) or if the transducer probe is in the air. If one or more of these conditions are met, a processor 102 transitions the imaging system from the normal operating state to the low frame rate state 204. The processors 102, 62 are programmed to periodically check for these conditions are move the imaging system to a lesser power state if any one of these conditions are detected”; and [0035], “In the power off state, the ultrasound imaging system is completely shut down and in one embodiment is only restarted upon a user pressing or activating the power button”. Note that detection the imaging system power state, and restarting it when needed to activate it, is mapped to the current claimed limitation).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Neumann to have the method of claim 2, wherein the initialization configuration instruction is determined by: determining the initialization configuration instruction in a situation where the imaging device is detected to be switched from a non-operational state to an operational state, the initialization configuration instruction being configured for instructing the server to construct a simulation model corresponding to the imaging device as taught by Imai in order to reduce the power consumption of such devices to increase the operating and standby time (See Imai: Fig. 1, and [0006], “The method proposed herein involves the generation of a 3D model of an organ using volume modeling. Unlike wire frame and surface modeling, volume modeling systems ensure that all surfaces meet properly and that the object is geometrically correct. Volume modeling simulates an object internally and externally. Volumic 3D models can be sectioned to reveal their internal features. When an object is built as a 3D model, cross sections of its internal structure can be rendered as if it were sliced”). Neumann teaches a method and system that may use the first AI model to reconstruct the human organ model based on the captured images and use a second AI model to generate the medical imaging configuration, and a third AI model to evaluate the second model, adjust the first model, and output the feedbacks to the respective first model and second model; while Imai teaches a system and method that may pre-program the imaging system in several power states and activate it when system is intended to be operation in order to save power. Therefore, it is obvious to one of ordinary skill in the art to modify Neumann by Imai to detect the power states of the imaging system and activate it when needed to same power consumption. The motivation to modify Neumann by Imai is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Claims 10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Neumann, etc. (US 20150242589 A1) in view of Hsieh, etc. (US 20190220975 A1), further in view of Amyot, etc. (US 20120128218 A1) and Verard, etc. (US 20140282008 A1).
Regarding claim 10, Neumann, Hsieh, and Amyot teach all the features with respect to claim 2 as outlined above. However, Neumann, modified by Hsieh and Amyot fails to explicitly disclose that the method of claim 1, wherein: the viewing device includes at least one of a visual reality (VR) device, an augmented reality (AR) device, a holographic imaging device, or a touch display screen; and the viewing device is configured to display the simulation image matching a first display mode based on a received first display mode configuration instruction.
However, Verard teaches that the method of claim 1, wherein:
the viewing device includes at least one of a visual reality (VR) device, an augmented reality (AR) device, a holographic imaging device, or a touch display screen (See Verard: Fig. 1, and [0033], “Referring now to the drawings in which like numerals represent the same or similar elements and initially to FIG. 1, a system 100 for generating and interacting with holographic images is illustratively shown in accordance with one embodiment. System 100 may include a workstation or console 112 from which a procedure is supervised and/or managed. Workstation 112 preferably includes one or more processors 114 and memory 116 for storing programs and applications. Memory 116 may store a holographic generation module 115 configured to render a holographic image on a display 158 or in-air depending on the application. The holographic generation module 115 codes image data to generate a three dimensional hologram. The coding may provide the hologram on a 2D display or in 3D media or 3D display. In one example, data from 3D imaging, e.g., computed tomography, ultrasound, magnetic resonance may be transformed into a hologram using spatial distribution and light intensity to render the hologram”); and
the viewing device is configured to display the simulation image matching a first display mode based on a received first display mode configuration instruction (See Verard: Fig. 1, and [0025], “In one embodiment, one could "touch" or otherwise interact with a specific region of interest in the 3D holographic display (e.g., using one or multiple fingers, virtual tools, or physical instruments being tracked within the same interaction space), and tissue characteristics would become available and displayed in the 3D hologram. Such "touch" can also be used to, e.g., rotate the virtual organ, zoom, tag points in 3D, draw a path and trajectory plan (e.g., for treatment, targeting, etc.), select critical zones to avoid, create alarms, and drop virtual objects (e.g., implants) in 3D in the displayed 3D anatomy”; and [0026], “Exemplary embodiments according to the present disclosure can also be used to facilitate a remote procedure (e.g., where the practitioner "acts" on the virtual organ and a robot simultaneously or subsequently performs the procedure on the actual organ), to practice a procedure before performing the actual procedure in a training or simulation setting, and/or to review/study/teach a procedure after it has been performed (e.g., through data recording, storage, and playback of the 3D holographic display and any associated multimodality signals relevant to the clinical procedure)”; and [0048], “Seed points 162 may be created and dropped into the 3D holographic display 158 or hologram 124 by touching (and/or tapping, holding, etc.) a portion of the display 158 or the hologram 124. The seed points 162 may be employed for, e.g., activation of virtual cameras which can provide individually customized viewing perspectives (e.g., orientation, zoom, resolution, etc.) which can be streamed (or otherwise transmitted) onto a separate high resolution 2D display 118”. Note that recording, playback, and rotating the virtual organs, zooming, etc. is changed the presentation of the displayed image, and this is mapped to the display mode adaptation).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Neumann to have the method of claim 1, wherein: the viewing device includes at least one of a visual reality (VR) device, an augmented reality (AR) device, a holographic imaging device, or a touch display screen; and the viewing device is configured to display the simulation image matching a first display mode based on a received first display mode configuration instruction as taught by Verard in order to provide a greater degree of human-data interaction during a procedure (See Verard: Fig. 1, and [0022], “In accordance with the present principles, systems, devices and methods are described which leverage holographic display technology for medical procedures. This can be done using 3D holographic technologies (e.g., in-air holograms) and real-time 3D input sensing methods such as optical shape sensing to provide a greater degree of human-data interaction during a procedure. Employing holographic technology with other technologies potentially simplifies procedure workflow, instrument selection, and manipulation within the anatomy of interest”). Neumann teaches a method and system that may use the first AI model to reconstruct the human organ model based on the captured images and use a second AI model to generate the medical imaging configuration, and a third AI model to evaluate the second model, adjust the first model, and output the feedbacks to the respective first model and second model; while Verard teaches a system and method that may include a holographic generation module configured to display a holographically rendered anatomical image. Therefore, it is obvious to one of ordinary skill in the art to modify Neumann by Verard to display a holographically rendered anatomical image in order to provide greater freedom of human-data interaction during a procedure. The motivation to modify Neumann by Verard is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 12, Neumann, Hsieh, and Amyot teach all the features with respect to claim 1 as outlined above. Further, Verard teaches that the method of claim 1, wherein the viewing device is further configured to display the simulation image matching a second display mode based on a received second display mode configuration instruction (See Verard: Fig. 1, and [0023], “In one exemplary embodiment, 3D holography may be used to fuse anatomical data with functional imaging and "sensing" information. A fourth dimension (e.g., time, color, texture, etc.) can be used to represent a dynamic 3D multimodality representation of the status of an object of interest (e.g., organ). A display can be in (near) real-time and use color-coded visual information and/or haptic feedback/tactile information, for example, to convey different effects of states of the holographically displayed object of interest. Such information can include morphological information about the target, functional information about the object of interest (e.g. flow, contractility, tissue biomechanical or chemical composition, voltage, temperature, pH, pO.sub.2, pCO.sub.2, etc.), or the measured changes in target properties due to interaction between the target and therapy being delivered. The exemplary 3D holographic display can be seen from (virtually) any angle/direction so that, e.g., multiple users can simultaneously interact with the same understanding and information”. Note that multi-modality representation and viewing from different angle, different visual endocing is mapped to the display mode).
Allowable Subject Matter
Claims 4 and 26 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, Neumann, etc. (US 20150242589 A1), Hsieh, etc. (US 20190220975 A1), and Amyot, etc. (US 20120128218 A1), do not teach the cited limitations of “the method of claim 1, wherein the simulation model includes a plurality of image constructing modules, each of the plurality of image constructing modules is configured with one or more construction parameters, the one or more construction parameters are used for generating the simulation image, and the generating a module parameter adjustment instruction based on one or more device state parameters transmitted by the imaging device comprises: generating the module parameter adjustment instruction in a situation where the one or more device state parameters transmitted by the imaging device are inconsistent with the one or more construction parameters corresponding to the each of the plurality of image construction modules; wherein the module parameter adjustment instruction is used for instructing the simulation model to adjust the one or more construction parameters corresponding to the each of the plurality of image construction modules.”
Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, Neumann, etc. (US 20150242589 A1), Hsieh, etc. (US 20190220975 A1), and Amyot, etc. (US 20120128218 A1), Imai, etc. (US 20200033461 A1), and Verard, etc. (US 20140282008 A1), do not teach the cited limitations of “the method of claim 10, wherein the simulation image matching the first display mode is displayed by: in a situation where the first display mode is a guidance mode, displaying the simulation image data corresponding to the simulation image transmitted by the server, and the identification content corresponding to the simulation image data; and in a situation where the first display mode is a consultation mode, displaying the simulation image data corresponding to the simulation image transmitted by the server.”
Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, Neumann, etc. (US 20150242589 A1), Hsieh, etc. (US 20190220975 A1), and Amyot, etc. (US 20120128218 A1), Imai, etc. (US 20200033461 A1), and Verard, etc. (US 20140282008 A1), do not teach the cited limitations of “the method of claim 12, wherein the displaying the simulation image matching the second display mode is displayed by: in a situation where the second display mode is a default display mode, displaying the simulation image data corresponding to the simulation image transmitted by the server; in a situation where the second display mode is a partial display mode, displaying the simulation image data corresponding to the simulation image transmitted by the server, and displaying, based on a simulation image data selection instruction, a portion of the identification content corresponding to selected simulation image data; and in a situation where the second display mode is a full display mode, displaying the simulation image data corresponding to the simulation image transmitted by the server, and the identification content corresponding to the simulation image data.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights