DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 10-13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Baram, etc. (US 20200029845 A1) in view of Severino (US 20160183824 A1).
Regarding claim 1, Baram teaches that a method (See Baram: Fig. 1, and [0027], “FIG. 1 is a schematic, pictorial illustration of a system 20 for electro-anatomical mapping, in accordance with an embodiment of the present invention. FIG. 1 depicts a physician 30 using an electro-anatomical catheter 40 to perform an electro-anatomical mapping of a cardiac chamber, such as a left atrium 45, of a heart 26 of a patient 28 laying on a table 29. By way of example, inset 25 shows catheter 40 as a PENTARY® mapping catheter (made by Biosense-Webster, Irvine, Calif.), which comprises one or more arms which may be mechanically flexible, each of which being coupled with one or more mapping electrodes. As seen, catheter 40 is fitted at the distal end of a shaft 22”) comprising:
by a control circuit (See Baram: Fig. 1, and [0035], “To extract surface 55, the processor typically runs an image processing software, such as software that extracts a partial surface of left atrium 45 from medical images, using the sphere intersection model. In an embodiment, a random sphere is created with a center close enough to the center of the imaged atrium volume to have a reasonable intersection. The processor applies the software to place the selected sphere in intersection with the imaged left atria volume while requiring (e.g., by minimizing the loss function) the network model to reconstruct the full input, i.e., the complete representation of the left atrium”):
accessing electroanatomic mapping (EAM) two-dimensional image information for a particular patient's heart (See Baram: Figs. 1-2A-B, and [0026], “The disclosed techniques reconstruct a realistic, and clinically valuable, shape of a cardiac chamber (e.g., a left atrium) from a sparse set of measured locations. By doing so, the disclosed technique and system may assist a physician in planning a proper treatment, such as a cardiac ablation. Since the disclosed techniques utilize only a sparse set of measurements, the mapping procedure may be shortened and simplified”; [0030], “During the procedure, a tracking system is used to track the respective locations of the mapping electrodes, such that each of the signals may be associated with the location at which the signal was acquired. For example, the Active Current Location (ACL) system, made by Biosense-Webster (Irvine, Calif.), which is described in U.S. Pat. No. 8,456,182, whose disclosure is incorporated herein by reference, may be used. In the ACL system, a processor estimates the respective locations of the electrodes based on impedances measured between each of the mapping-electrodes, and a plurality of surface-electrodes (not shown) that are coupled to the skin of patient 28. Processor 38 calculates a data-set of estimated locations along one or more paths of catheter 40 inside left atrium 45”; and [0038], “In order to reconstruct a realistic volume of left atrium 45 from sparsely measured locations, the disclosed neural network computation technique uses a loss function (i.e., a neural network model), G, which includes a regularization-function, F, that comprises smoothing spatial weights, in addition to a cross-entropy loss term L. In some embodiments, L(x,z) is a logarithmic norm function, based on training, to achieve a “best fit” of z values to the measured locations x”. Note that the sparse electrode locations in the left atrium are mapped to the particular patient’s heart);
generating a three-dimensional EAM presentation of the particular patient's heart as a function of the EAM two-dimensional image information to provide a generated three-dimensional EAM presentation of the particular patient's heart (See Baram: Figs. 1-5, and [0019], “A cardiac chamber, such as a left atrium, has a geometrically complex shape that may be electro-anatomically mapped in a partial manner during a mapping procedure, such as using catheter-based anatomical mapping”; [0047], To reconstruct object 60 from sparse data, processor applies an input layer 61 an autoencoder module 65 comprising a neural network module 62 and a regularization module 63. Processor 38 applies autoencoder 65 using a given number of hidden layers of the neural network model, and a given number of voxels to represent input layer 61. Processor 38 generates a 3D output layer 64 that is the reconstructed left atrium 66 (e.g., reconstructed left atrium 80 of FIG. 5) having a same given number of voxels as assigned to input layer 61, but one that comprises a learned left atrium shape that the sparse data best fit into”).
However, Baram fails to explicitly disclose that electroanatomic mapping (EAM) two-dimensional image information.
However, Severino teaches that electroanatomic mapping (EAM) two-dimensional image information (See Severino: Figs. 4A-B, and [0010], “Additionally, there is also a growing variety of special-purpose electrode catheters available. They can produce electrogram patterns that are much more complex and therefore difficult to interpret by electrogram pattern alone, as shown in FIGS. 4A and 4B”; and Figs. 21A-E, and [0126], “In one embodiment, the assigned relative times T(S×m) for each shaft subsection is weighted according to its location between the earlier- and later-acquiring adjacent poles, and the weighting is applied linearly and dependent on the plurality of subsections between the adjacent poles. An example of the processing of block 209 designating and block 210 (applying Eqns. 2, 3 and 4) assigning relative times for the shaft subsections is shown in FIG. 18E, along with an example of the processing of blocks 206, 207, and 205 for the poles. Per the processing of blocks 214 and 215 correlating both the relative times for the poles and the shaft sections, a display sequence as shown in FIGS. 21A-21I is generated by the system and method of the present invention”. Note that the 2D electrogram patterns are mapped to the 2D image information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Baram to have electroanatomic mapping (EAM) two-dimensional image information as taught by Severino in order to avoid overlapping sequences of electrode acquisitions (See Severino: Fig. 22, and [0114], “Because the entire sequence of electrode acquisition may have a duration on the order of several milliseconds and thus be imperceptible to the human eye, the animation speed may be adjusted. Query 216 of FIG. 17 asks whether a user wishes to adjust the display sequence, for example, by selecting a time scale at which the animation is displayed, per block 218. If no, the processing ends at block 220. If yes, block 218 allows a user to make a selection, whereby, for example, the duration of the animation is increased by a selected factor N by multiplying each relative time T(i) by N, and a re-correlation is performed in block 214 in accordance with the selected time scale. An example of the processing of blocks 214, 216 and 218 is shown in FIG. 18C. As another example in lieu or in addition to block 218, block 219 allows the user to adjust or limit the ratio of visual representations to actual cardiac cycles in order to facilitate ease of viewing and avoid overlapping sequences of electrode acquisitions. The user may select which electrode sequences are displayed, including whether the animation includes the acquisition sequence of every “nth” cardiac cycle. Upon a selection by the user, a re-correlation is performed in block 214”). Baram teaches a method and system that may reconstruct the left atrium of the patients based on the sparse location measurements using electro-anatomic mapping techniques with neural neatwork modeling; while Severino teaches a system and method that may process the electrode signals, generate a sequence of electrode signal acquisitions, and generate a visual representation of the sequence of electrode signal acquisitions to generate a visual representation with a graphical image of the electrodes and to present it to the patients. Therefore, it is obvious to one of ordinary skill in the art to modify Baram by Severino to process the 2D EAM images and present the reconstructed 3D heart presentation to the users. The motivation to modify Baram by Severino is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Baram and Severino teach all the features with respect to claim 1 as outlined above. Further, Severino teaches that the method of claim 1 wherein accessing EAM two-dimensional image information for the particular patient's heart comprises accessing two-dimensional screen shots (See Severino: Figs. 5-6, and [0020], “Additionally, a completed LAT map can be visualized as a “propagation map” where the activation sequence on the map is played by the mapping system as an animation, showing the spread or propagation of electrical activation across the mapped region of interest each time it repeats. This can be a very helpful, dynamic alternative to visually following the rainbow scale of activation around a static LAT map where very slight, yet potentially important changes in color shade, may be missed. FIG. 6 shows a series of screen captures from a propagation map animation of the focal activation sequence shown in FIG. 5. In the animation, the wave in color red of depolarization moves in time across the chamber shown in color blue. Generally, the animations loop continually so the wave can be studied again once it plays through”).
Regarding claim 3, Baram and Severino teach all the features with respect to claim 2 as outlined above. Further, Severino teaches that the method of claim 2 wherein accessing two-dimensional screen shots comprises accessing at least two different two-dimensional screen shots (See Severino: Figs. 5-6, and [0020], “Additionally, a completed LAT map can be visualized as a “propagation map” where the activation sequence on the map is played by the mapping system as an animation, showing the spread or propagation of electrical activation across the mapped region of interest each time it repeats. This can be a very helpful, dynamic alternative to visually following the rainbow scale of activation around a static LAT map where very slight, yet potentially important changes in color shade, may be missed. FIG. 6 shows a series of screen captures from a propagation map animation of the focal activation sequence shown in FIG. 5. In the animation, the wave in color red of depolarization moves in time across the chamber shown in color blue. Generally, the animations loop continually so the wave can be studied again once it plays through”. Note that Fig. 6 shows 9 screen captures).
Regarding claim 10, Baram and Severino teach all the features with respect to claim 1 as outlined above. Further, Severino teaches that the method of claim 1 further comprising:
generating a treatment plan to treat a condition of the particular patient's heart using the generated three-dimensional EAM presentation of the particular patient's heart (See Severino: Figs. 2-7, and [0028], “For the simpler arrhythmias, the electrophysiologist may choose not to remap but merely refer to the electrograms of properly positioned multipolar catheters (for example, see FIGS. 2B and 3B), which, as mentioned earlier, are typically displayed on a recording device during an ablation procedure to provide additional data for use by the electrophysiologist. Electrograms may be particularly informative for those regions or chambers of the heart where there are well-established, standard catheter positions as well as established ablation patterns. An atrial flutter ablation procedure is one of the simplest examples of this. As briefly mentioned earlier, reentrant signals of atrial flutter in the right atrium typically have a circuitous path that is clockwise or counterclockwise around the tricuspid valve annulus TVA. FIG. 7A shows a map of clockwise atrial flutter created using FAM and a dual-purpose, sensor-based mapping and ablation catheter. The red-to-purple color pattern in this map can be traced clockwise around the valve (center circular cutout with thin green border) from the red area in the upper corner all the way around in a loop to the purple area returning to the starting point (CARTO 3 places the brown “early meets late line” into the map automatically between red and purple points). Only one cardiac sequence is described by the map; in reality the wave of depolarization continues around and around in a continuous loop around the TVA. In FIG. 7A, three catheters are visualized. Catheters for this procedure typically include a nonmagnetic, current-based sensing “Duo-deca” multipolar catheter (in green) that enters the right atrium RA from the IVC and is generally positioned in a loop just outside the TVA. Its electrograms, therefore, help describe how the electrical activation is moving around the tricuspid valve. Longer versions of this catheter (actually, ones with more widely-spaced electrode pairs), like the one pictured, can extend across the floor of right atrium (the cavotricuspid isthmus) and into the coronary sinus ostium. A properly positioned Duo-Deca catheter produces a very distinct “slanted” electrogram patterns in atrial flutter (see for example, FIG. 2B). The direction of the slant indicates whether it is clockwise or counterclockwise atrial flutter (for example, clockwise in FIG. 2B). Also visible in this map is the distal tip of a nonmagnetic, current-based HIS catheter (in green) protruding through the TCV from the right atrium RA into the right ventricle RV, and, of course, the magnetic sensing mapping and ablation catheter (in white), shown protruding from the IVC at the cavotricuspid istmus”);
administering treatment to the particular patient's heart using the treatment plan (See Severino: Figs. 12A-B, and [0098], “The present invention can perhaps be better understood by making reference to the drawings, starting with FIG. 14A is a pictorial illustration of a system S for performing exemplary catheterization procedures on a heart 12 of a living subject or patient 13, which is constructed and operative in accordance with a disclosed embodiment of the invention. The system comprises a catheter 14, which is percutaneously inserted by an electrophysiologist or operator 16 through the patient's vascular system into a chamber or vascular structure of the heart 12. The catheter 14 has a distal tip carrying a plurality of electrodes, and a control handle by which the operator 16 can manipulate to steer and deflect the catheter”).
Regarding claim 11, Baram and Severino teach all the features with respect to claim 1 as outlined above. Further, Baram and Severino teach that an apparatus (See Baram: Fig. 1, and [0027], “FIG. 1 is a schematic, pictorial illustration of a system 20 for electro-anatomical mapping, in accordance with an embodiment of the present invention. FIG. 1 depicts a physician 30 using an electro-anatomical catheter 40 to perform an electro-anatomical mapping of a cardiac chamber, such as a left atrium 45, of a heart 26 of a patient 28 laying on a table 29. By way of example, inset 25 shows catheter 40 as a PENTARY® mapping catheter (made by Biosense-Webster, Irvine, Calif.), which comprises one or more arms which may be mechanically flexible, each of which being coupled with one or more mapping electrodes. As seen, catheter 40 is fitted at the distal end of a shaft 22”) comprising:
a control circuit configured (See Baram: Fig. 1, and [0035], “To extract surface 55, the processor typically runs an image processing software, such as software that extracts a partial surface of left atrium 45 from medical images, using the sphere intersection model. In an embodiment, a random sphere is created with a center close enough to the center of the imaged atrium volume to have a reasonable intersection. The processor applies the software to place the selected sphere in intersection with the imaged left atria volume while requiring (e.g., by minimizing the loss function) the network model to reconstruct the full input, i.e., the complete representation of the left atrium”) to:
access electroanatomic mapping (EAM) two-dimensional image information (See Severino: Figs. 4A-B, and [0010], “Additionally, there is also a growing variety of special-purpose electrode catheters available. They can produce electrogram patterns that are much more complex and therefore difficult to interpret by electrogram pattern alone, as shown in FIGS. 4A and 4B”; and Figs. 21A-E, and [0126], “In one embodiment, the assigned relative times T(S×m) for each shaft subsection is weighted according to its location between the earlier- and later-acquiring adjacent poles, and the weighting is applied linearly and dependent on the plurality of subsections between the adjacent poles. An example of the processing of block 209 designating and block 210 (applying Eqns. 2, 3 and 4) assigning relative times for the shaft subsections is shown in FIG. 18E, along with an example of the processing of blocks 206, 207, and 205 for the poles. Per the processing of blocks 214 and 215 correlating both the relative times for the poles and the shaft sections, a display sequence as shown in FIGS. 21A-21I is generated by the system and method of the present invention”. Note that the 2D electrogram patterns are mapped to the 2D image information) for a particular patient's heart (See Baram: Figs. 1-2A-B, and [0026], “The disclosed techniques reconstruct a realistic, and clinically valuable, shape of a cardiac chamber (e.g., a left atrium) from a sparse set of measured locations. By doing so, the disclosed technique and system may assist a physician in planning a proper treatment, such as a cardiac ablation. Since the disclosed techniques utilize only a sparse set of measurements, the mapping procedure may be shortened and simplified”; [0030], “During the procedure, a tracking system is used to track the respective locations of the mapping electrodes, such that each of the signals may be associated with the location at which the signal was acquired. For example, the Active Current Location (ACL) system, made by Biosense-Webster (Irvine, Calif.), which is described in U.S. Pat. No. 8,456,182, whose disclosure is incorporated herein by reference, may be used. In the ACL system, a processor estimates the respective locations of the electrodes based on impedances measured between each of the mapping-electrodes, and a plurality of surface-electrodes (not shown) that are coupled to the skin of patient 28. Processor 38 calculates a data-set of estimated locations along one or more paths of catheter 40 inside left atrium 45”; and [0038], “In order to reconstruct a realistic volume of left atrium 45 from sparsely measured locations, the disclosed neural network computation technique uses a loss function (i.e., a neural network model), G, which includes a regularization-function, F, that comprises smoothing spatial weights, in addition to a cross-entropy loss term L. In some embodiments, L(x,z) is a logarithmic norm function, based on training, to achieve a “best fit” of z values to the measured locations x”. Note that the sparse electrode locations in the left atrium are mapped to the particular patient’s heart);
generate a three-dimensional EAM presentation of the particular patient's heart as a function of the EAM two-dimensional image information to provide a generated three-dimensional EAM presentation of the particular patient's heart (See Baram: Figs. 1-5, and [0019], “A cardiac chamber, such as a left atrium, has a geometrically complex shape that may be electro-anatomically mapped in a partial manner during a mapping procedure, such as using catheter-based anatomical mapping”; [0047], To reconstruct object 60 from sparse data, processor applies an input layer 61 an autoencoder module 65 comprising a neural network module 62 and a regularization module 63. Processor 38 applies autoencoder 65 using a given number of hidden layers of the neural network model, and a given number of voxels to represent input layer 61. Processor 38 generates a 3D output layer 64 that is the reconstructed left atrium 66 (e.g., reconstructed left atrium 80 of FIG. 5) having a same given number of voxels as assigned to input layer 61, but one that comprises a learned left atrium shape that the sparse data best fit into”).
Regarding claim 12, Baram and Severino teach all the features with respect to claim 11 as outlined above. Further, Severino teaches that the apparatus of claim 11 wherein the control circuit is configured to access EAM two-dimensional image information for the particular patient's heart by accessing two-dimensional screen shots (See Severino: Figs. 5-6, and [0020], “Additionally, a completed LAT map can be visualized as a “propagation map” where the activation sequence on the map is played by the mapping system as an animation, showing the spread or propagation of electrical activation across the mapped region of interest each time it repeats. This can be a very helpful, dynamic alternative to visually following the rainbow scale of activation around a static LAT map where very slight, yet potentially important changes in color shade, may be missed. FIG. 6 shows a series of screen captures from a propagation map animation of the focal activation sequence shown in FIG. 5. In the animation, the wave in color red of depolarization moves in time across the chamber shown in color blue. Generally, the animations loop continually so the wave can be studied again once it plays through”).
Regarding claim 13, Baram and Severino teach all the features with respect to claim 12 as outlined above. Further, Severino teaches that the apparatus of claim 12 wherein the control circuit is configured to access two-dimensional screen shots by accessing at least two different two-dimensional screen shots (See Severino: Figs. 5-6, and [0020], “Additionally, a completed LAT map can be visualized as a “propagation map” where the activation sequence on the map is played by the mapping system as an animation, showing the spread or propagation of electrical activation across the mapped region of interest each time it repeats. This can be a very helpful, dynamic alternative to visually following the rainbow scale of activation around a static LAT map where very slight, yet potentially important changes in color shade, may be missed. FIG. 6 shows a series of screen captures from a propagation map animation of the focal activation sequence shown in FIG. 5. In the animation, the wave in color red of depolarization moves in time across the chamber shown in color blue. Generally, the animations loop continually so the wave can be studied again once it plays through”. Note that Fig. 6 shows 9 screen captures).
Regarding claim 20, Baram and Severino teach all the features with respect to claim 11 as outlined above. Further, Severino teaches that the apparatus of claim 11 wherein the control circuit is further configured to:
generate a treatment plan to treat a condition of the particular patient's heart using the generated three-dimensional EAM presentation of the particular patient's heart (See Severino: Figs. 2-7, and [0028], “For the simpler arrhythmias, the electrophysiologist may choose not to remap but merely refer to the electrograms of properly positioned multipolar catheters (for example, see FIGS. 2B and 3B), which, as mentioned earlier, are typically displayed on a recording device during an ablation procedure to provide additional data for use by the electrophysiologist. Electrograms may be particularly informative for those regions or chambers of the heart where there are well-established, standard catheter positions as well as established ablation patterns. An atrial flutter ablation procedure is one of the simplest examples of this. As briefly mentioned earlier, reentrant signals of atrial flutter in the right atrium typically have a circuitous path that is clockwise or counterclockwise around the tricuspid valve annulus TVA. FIG. 7A shows a map of clockwise atrial flutter created using FAM and a dual-purpose, sensor-based mapping and ablation catheter. The red-to-purple color pattern in this map can be traced clockwise around the valve (center circular cutout with thin green border) from the red area in the upper corner all the way around in a loop to the purple area returning to the starting point (CARTO 3 places the brown “early meets late line” into the map automatically between red and purple points). Only one cardiac sequence is described by the map; in reality the wave of depolarization continues around and around in a continuous loop around the TVA. In FIG. 7A, three catheters are visualized. Catheters for this procedure typically include a nonmagnetic, current-based sensing “Duo-deca” multipolar catheter (in green) that enters the right atrium RA from the IVC and is generally positioned in a loop just outside the TVA. Its electrograms, therefore, help describe how the electrical activation is moving around the tricuspid valve. Longer versions of this catheter (actually, ones with more widely-spaced electrode pairs), like the one pictured, can extend across the floor of right atrium (the cavotricuspid isthmus) and into the coronary sinus ostium. A properly positioned Duo-Deca catheter produces a very distinct “slanted” electrogram patterns in atrial flutter (see for example, FIG. 2B). The direction of the slant indicates whether it is clockwise or counterclockwise atrial flutter (for example, clockwise in FIG. 2B). Also visible in this map is the distal tip of a nonmagnetic, current-based HIS catheter (in green) protruding through the TCV from the right atrium RA into the right ventricle RV, and, of course, the magnetic sensing mapping and ablation catheter (in white), shown protruding from the IVC at the cavotricuspid istmus”);
administer treatment to the particular patient's heart using the treatment plan (See Severino: Figs. 12A-B, and [0098], “The present invention can perhaps be better understood by making reference to the drawings, starting with FIG. 14A is a pictorial illustration of a system S for performing exemplary catheterization procedures on a heart 12 of a living subject or patient 13, which is constructed and operative in accordance with a disclosed embodiment of the invention. The system comprises a catheter 14, which is percutaneously inserted by an electrophysiologist or operator 16 through the patient's vascular system into a chamber or vascular structure of the heart 12. The catheter 14 has a distal tip carrying a plurality of electrodes, and a control handle by which the operator 16 can manipulate to steer and deflect the catheter”).
Claims 4, 7, 14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Baram, etc. (US 20200029845 A1) in view of Severino (US 20160183824 A1), further in view of Barve, etc. (US 20250245829 A1).
Regarding claim 4, Baram and Severino teach all the features with respect to claim 1 as outlined above. Further, Baram teaches that the method of claim 1 further comprising:
inputting information corresponding to the EAM two-dimensional image information to a neural network that outputs a texture map and an initial three-dimensional mesh with texture coordinates that both correspond to the particular patient's heart (See Baram: Figs. 3-4, and [0038], “In order to reconstruct a realistic volume of left atrium 45 from sparsely measured locations, the disclosed neural network computation technique uses a loss function (i.e., a neural network model), G, which includes a regularization-function, F, that comprises smoothing spatial weights, in addition to a cross-entropy loss term L. In some embodiments, L(x,z) is a logarithmic norm function, based on training, to achieve a “best fit” of z values to the measured locations x”; [0049], “Training phase 71 begins with processor 38 receiving example representations of geometrical shapes of left atria, at a database uploading step 74. Next, processor 38 runs the disclosed neural network model over the database of example representations, so as to train the network model, at a neural network training step 76. Finally, processor 38 stores the parameters of the trained neural network model in memory 41, or in a disk, or keep the model such way that the model can be directly loaded later during the procedure, at a storing step 7”; and [0050], “In modeling phase 72, electro-anatomical system 20 measures a set of locations in left atrium 45, as described above, in a locations acquisition step 84. Next, processor 38 runs the disclosed trained neural network model from step 76 over the measured set of locations, at a neural network model running step 86. Finally, processor 38 produces a three-dimensional model of left atrium 45, at a neural network modeling step 88”).
However, Baram, modified by Severino, fails to explicitly disclose that a texture map and an initial three-dimensional mesh with texture coordinates.
However, Barve teaches that a texture map and an initial three-dimensional mesh with texture coordinates (See Barve: Fig. 1, and [0071], “Still referring to FIG. 1, in some embodiments, SSM may be used to refine a mesh produced using a 3D model estimate. For example, a machine learning model such as a deep neural network may be used to produce a voxel grid estimate of patient's organ based on ICE frames of the patient's organ, and SSM may be used to refine this estimate to produce a 3D mesh of patient's organ. In some embodiments, this may include deforming a template of patient's organ to arrive at a most probable shape that matches a DNN estimate. Additionally, or alternatively, a machine learning model may be trained to directly generate parameters of a statistical shape model”; and [0089], “Still referring to FIG. 1, processor 104 is configured to overlay map 164 onto heart model. In some embodiments, the overlay may be placed on 3D model 156 and go through a refinement process as described above. Overlaying map 164 on a model may include implementing spatial alignment methods, texture mapping techniques wherein the color information from the heat map is mapped onto the vertices or faces of the 3D model, shader programs that define how the heat map values influence the final appearance of the 3D model, visualization software or programming libraries that support 3D rendering and overlay capabilities, interactivity visualization, quality control methods, and the like. For example, texture mapping may include UV Mapping wherein each point on the surface of a 3D model is associated with a set of texture coordinates often denoted as U and V. These coordinates are analogous to the X and Y coordinates on a 2D image. UV mapping establishes the correspondence between points on the 3D model and pixels on the 2D texture. In another example, interactive visualization may create visual representations of data that users can interact with and manipulate. This approach allows users to explore and analyze data dynamically, gaining insights through direct engagement with the visual representation. For example, mouse interactivity may allow users to interact with visual elements using mouse actions, such as hovering over data points for additional information, clicking to drill down into details, or dragging to pan and zoom. Filtering and Selection capabilities may allow a user to filter data based on specific criteria or select subsets of data for closer examination. This is particularly useful when dealing with large datasets. Spatial Exploration may allow users to zoom in to explore details or pan to navigate across the space”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Baram to have a texture map and an initial three-dimensional mesh with texture coordinates as taught by Barve in order to detect accurately detection ensures that the device effectively prevents blood flow into the LAA, thus reducing the risk of complications (See Barve: Fig. 1, and [0091], “Still referring to FIG. 1, further, 3D model may be used in the placement and sizing of medical devices, such as, without limitation, left atrial appendage occlusion (LAAO) device. As a non-limiting example, ICE imaging may be used to capture set of images of a patient's organ e.g., a heart (specifically focusing on the LA and LAA). 3D model created using such set of images may serve as a basis for planning and executing the placement of the LAAO device. In some cases, the 3D model may allow clinicians to visualize the exact structure and dimensions of the modeled LA and LAA in order to determine an appropriate size of the LAAO device to ensure optimal fit and function. For example, and without limitation, using the 3D model, clinicians may plan the precise placement of the LAAO device according to one or more anatomical landmarks on the 3D model. After the initial placement, another imaging session may be conducted to verify the positioning and fit of the LAAO device. In some cases, the two imaging session may be performed via different imaging techniques; for instance, and without limitation, second set of images may include one or more CT images of the heart. In some cases, 3D model may be used in detecting any potential leakage around the device. Accurate detection may ensure that the device effectively prevents blood flow into the LAA thereby reducing the risk of complications”). Baram teaches a method and system that may reconstruct the left atrium of the patients based on the sparse location measurements using electro-anatomic mapping techniques with neural neatwork modeling; while Barve teaches a system and method that may generate 3D model of the patient’s organ using neural network model based on the input 2D images and refining the 3D model with textures and mesh iterative feedback and refining process. Therefore, it is obvious to one of ordinary skill in the art to modify Baram by Barve to refine the 3D heart representation model with 3D mesh with texture coordinate and texture maps. The motivation to modify Baram by Barve is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 7, Baram, Severino, and Barve teach all the features with respect to claim 4 as outlined above. Further, Baram and Severino teach that the method of claim 4 further comprising:
generating an approximate three-dimensional EAM model for the particular patient's heart (See Baram: Figs. 3-5, and [0050], “In modeling phase 72, electro-anatomical system 20 measures a set of locations in left atrium 45, as described above, in a locations acquisition step 84. Next, processor 38 runs the disclosed trained neural network model from step 76 over the measured set of locations, at a neural network model running step 86. Finally, processor 38 produces a three-dimensional model of left atrium 45, at a neural network modeling step 88”) as a function of the texture map and the initial three-dimensional mesh (See Severino: Fig. 1, and [0066], “With continued reference to FIG. 1, processor 104 may use a statistical shape model (SSM) to generate and/or iteratively refine a 3D model 156 based on a set of shape parameters. As used herein, a “heart model” is a 3D representation of patient's organ. In some cases, 3D model 156 may be generated through a direct 3D reconstruction from a series of (2D) ICE images. In a non-limiting example, set of images 112 may include a plurality of ICE images captured from different angles and positions within the heart. Processor 104 may be configured to apply one or more 3D reconstruction algorithms, such as without limitation, marching cubes, contour detection and segmentation, active contour models, and/or the like to create a coherent 3D representation e.g., 3D model 156 of patient's organ 116. In some cases, such direct 3D reconstruction may leverage the inherent spatial information within set of images 112, providing a direct and intuitive way to model the 3D model 156 of the heart's structure. In a further embodiment, generic 3D modeling techniques may be applied to create the initial 3D model. In some cases, generic 3D modeling techniques may include surface modeling, solid modeling, or parametric modeling, among others. As an ordinary person skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various 3D reconstruction algorithms that may be used by processor 104 to generate 3D model 156 of patient's organ 116”; and {0089}, “Still referring to FIG. 1, processor 104 is configured to overlay map 164 onto heart model. In some embodiments, the overlay may be placed on 3D model 156 and go through a refinement process as described above. Overlaying map 164 on a model may include implementing spatial alignment methods, texture mapping techniques wherein the color information from the heat map is mapped onto the vertices or faces of the 3D model, shader programs that define how the heat map values influence the final appearance of the 3D model, visualization software or programming libraries that support 3D rendering and overlay capabilities, interactivity visualization, quality control methods, and the like. For example, texture mapping may include UV Mapping wherein each point on the surface of a 3D model is associated with a set of texture coordinates often denoted as U and V. These coordinates are analogous to the X and Y coordinates on a 2D image. UV mapping establishes the correspondence between points on the 3D model and pixels on the 2D texture”).
Regarding claim 14, Baram and Severino teach all the features with respect to claim 11 as outlined above. Further, Baram teaches that the apparatus of claim 11 wherein the control circuit is further configured to: input information corresponding to the EAM two-dimensional image information to a neural network that outputs a texture map and an initial three-dimensional mesh with texture coordinates (See Barve: Fig. 1, and [0071], “Still referring to FIG. 1, in some embodiments, SSM may be used to refine a mesh produced using a 3D model estimate. For example, a machine learning model such as a deep neural network may be used to produce a voxel grid estimate of patient's organ based on ICE frames of the patient's organ, and SSM may be used to refine this estimate to produce a 3D mesh of patient's organ. In some embodiments, this may include deforming a template of patient's organ to arrive at a most probable shape that matches a DNN estimate. Additionally, or alternatively, a machine learning model may be trained to directly generate parameters of a statistical shape model”; and [0089], “Still referring to FIG. 1, processor 104 is configured to overlay map 164 onto heart model. In some embodiments, the overlay may be placed on 3D model 156 and go through a refinement process as described above. Overlaying map 164 on a model may include implementing spatial alignment methods, texture mapping techniques wherein the color information from the heat map is mapped onto the vertices or faces of the 3D model, shader programs that define how the heat map values influence the final appearance of the 3D model, visualization software or programming libraries that support 3D rendering and overlay capabilities, interactivity visualization, quality control methods, and the like. For example, texture mapping may include UV Mapping wherein each point on the surface of a 3D model is associated with a set of texture coordinates often denoted as U and V. These coordinates are analogous to the X and Y coordinates on a 2D image. UV mapping establishes the correspondence between points on the 3D model and pixels on the 2D texture. In another example, interactive visualization may create visual representations of data that users can interact with and manipulate. This approach allows users to explore and analyze data dynamically, gaining insights through direct engagement with the visual representation. For example, mouse interactivity may allow users to interact with visual elements using mouse actions, such as hovering over data points for additional information, clicking to drill down into details, or dragging to pan and zoom. Filtering and Selection capabilities may allow a user to filter data based on specific criteria or select subsets of data for closer examination. This is particularly useful when dealing with large datasets. Spatial Exploration may allow users to zoom in to explore details or pan to navigate across the space”) that both correspond to the particular patient's heart (See Baram: Figs. 3-4, and [0038], “In order to reconstruct a realistic volume of left atrium 45 from sparsely measured locations, the disclosed neural network computation technique uses a loss function (i.e., a neural network model), G, which includes a regularization-function, F, that comprises smoothing spatial weights, in addition to a cross-entropy loss term L. In some embodiments, L(x,z) is a logarithmic norm function, based on training, to achieve a “best fit” of z values to the measured locations x”; [0049], “Training phase 71 begins with processor 38 receiving example representations of geometrical shapes of left atria, at a database uploading step 74. Next, processor 38 runs the disclosed neural network model over the database of example representations, so as to train the network model, at a neural network training step 76. Finally, processor 38 stores the parameters of the trained neural network model in memory 41, or in a disk, or keep the model such way that the model can be directly loaded later during the procedure, at a storing step 7”; and [0050], “In modeling phase 72, electro-anatomical system 20 measures a set of locations in left atrium 45, as described above, in a locations acquisition step 84. Next, processor 38 runs the disclosed trained neural network model from step 76 over the measured set of locations, at a neural network model running step 86. Finally, processor 38 produces a three-dimensional model of left atrium 45, at a neural network modeling step 88”).
Regarding claim 17, Baram and Severino teach all the features with respect to claim 14 as outlined above. Further, Baram and Severino teach that the apparatus of claim 14 wherein the control circuit is further configured to: generate an approximate three-dimensional EAM model for the particular patient's heart (See Baram: Figs. 3-5, and [0050], “In modeling phase 72, electro-anatomical system 20 measures a set of locations in left atrium 45, as described above, in a locations acquisition step 84. Next, processor 38 runs the disclosed trained neural network model from step 76 over the measured set of locations, at a neural network model running step 86. Finally, processor 38 produces a three-dimensional model of left atrium 45, at a neural network modeling step 88”) as a function of the texture map and the initial three-dimensional mesh (See Severino: Fig. 1, and [0066], “With continued reference to FIG. 1, processor 104 may use a statistical shape model (SSM) to generate and/or iteratively refine a 3D model 156 based on a set of shape parameters. As used herein, a “heart model” is a 3D representation of patient's organ. In some cases, 3D model 156 may be generated through a direct 3D reconstruction from a series of (2D) ICE images. In a non-limiting example, set of images 112 may include a plurality of ICE images captured from different angles and positions within the heart. Processor 104 may be configured to apply one or more 3D reconstruction algorithms, such as without limitation, marching cubes, contour detection and segmentation, active contour models, and/or the like to create a coherent 3D representation e.g., 3D model 156 of patient's organ 116. In some cases, such direct 3D reconstruction may leverage the inherent spatial information within set of images 112, providing a direct and intuitive way to model the 3D model 156 of the heart's structure. In a further embodiment, generic 3D modeling techniques may be applied to create the initial 3D model. In some cases, generic 3D modeling techniques may include surface modeling, solid modeling, or parametric modeling, among others. As an ordinary person skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various 3D reconstruction algorithms that may be used by processor 104 to generate 3D model 156 of patient's organ 116”; and {0089}, “Still referring to FIG. 1, processor 104 is configured to overlay map 164 onto heart model. In some embodiments, the overlay may be placed on 3D model 156 and go through a refinement process as described above. Overlaying map 164 on a model may include implementing spatial alignment methods, texture mapping techniques wherein the color information from the heat map is mapped onto the vertices or faces of the 3D model, shader programs that define how the heat map values influence the final appearance of the 3D model, visualization software or programming libraries that support 3D rendering and overlay capabilities, interactivity visualization, quality control methods, and the like. For example, texture mapping may include UV Mapping wherein each point on the surface of a 3D model is associated with a set of texture coordinates often denoted as U and V. These coordinates are analogous to the X and Y coordinates on a 2D image. UV mapping establishes the correspondence between points on the 3D model and pixels on the 2D texture”).
Claims 5-6 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Baram, etc. (US 20200029845 A1) in view of Severino (US 20160183824 A1), further in view of Barve, etc. (US 20250245829 A1) and Villongco (US 20190333643 A1).
Regarding claim 5, Baram, Severino, and Barve teach all the features with respect to claim 4 as outlined above. However, Baram, modified by Severino and Barve, fails to explicitly disclose that the method of claim 4 wherein the neural network comprises a trained neural network that has been trained with a training corpus comprising a plurality of synthetic EAM images that were generated using a variety of different anatomical and electrophysiological properties.
However, Villongco teaches that the method of claim 4 wherein the neural network comprises a trained neural network that has been trained with a training corpus comprising a plurality of synthetic EAM images that were generated using a variety of different anatomical and electrophysiological properties (See Villongco: Fig. 2, and [0061], “FIG. 2 is a flow diagram that illustrates the overall processing of generating a classifier by the MLMO system in some embodiments. A generate classifier component 200 is executed to generate a classifier. In block 201, the component accesses the computational model to be used to run the simulations. In block 202, the component selects the next source configuration (i.e., parameter set) to be used in a simulation. In decision block 203, if all the source configurations have already been selected, then the component continues at block 205, else the component continues at block 204. In block 204, the component runs the simulation using the selected source configuration to generate an EM output for the simulation and then loops to block 202 to select the next source configuration. In block 205, the component selects the next EM output that was generated by a simulation. In decision block 206, if all the EM outputs have already been selected, then the component continues at block 210, else the component continues at block 207. In block 207, the component derives the EM data from the EM output. For example, the EM output may be a collection of EM meshes, and the EM data may be an ECG or a VCG derived from the electromagnetic values of the EM mesh. In some embodiments, the component may in addition identify cycles (periodic intervals of arrhythmic activity) within the ECG or VCG. A cycle may be delimited by successive crossings from a negative voltage to a positive voltage (“positive crossings”) or successive crossings from a positive voltage to a negative voltage (“negative crossings”) with respect to a spatial direction or set of directions comprising a reference frame or set of reference frames. A reference frame may coincide with anatomical axes (e.g., left-to-right with x, superior-to-inferior with y, anterior-to-posterior with z), imaging axes (e.g., CT, MR, or x-ray coordinate frames), body-surface lead vectors, principal axes computed by principal component analysis of measured or simulated EM source configurations and outputs, or user-defined directions of interest. For example, a three-second VCG may have three cycles, and each cycle may be delimited by the times of the positive crossings along the x-axis. Alternatively, the cycles may be delimited by crossings along the y-axis or z-axis. In addition, cycles may be defined by negative crossings. Thus, in some embodiments, the component may generate training data from a single VCG based on various cycle definitions that are various combinations of positive crossings and negative crossings with the cycles for all the axes being defined by crossings on one of the x-axis, y-axis, and z-axis or the cycles for each defined by crossings on that axis. Moreover, the training data may include cycles identified based on all possible cycle definitions or a subset of the cycle definition. For example, the training data may include, for each axis, a cycle defined by positive crossings of the x-axis, negative crossings of the y-axis, and positive crossings of that axis itself. Cycles definitions may also be defined by the timing of electrical events derived from the values stored in the EM mesh. For example, a point or set of points in the mesh may periodically cross voltage thresholds signifying electrical activation and deactivation. Thus, a cycle may be defined by activation-deactivation, or successive activation-activation or deactivation-deactivation intervals corresponding to a point or set of points within the mesh. The resulting timings of these intervals can be co-localized to the ECG or VCG for cycles identification. In block 208, the component labels the EM data based on the source configuration (e.g., a source location). When cycles are identified, the component may label each cycle with the same label. For example, the component may label the identified cycles with the same rotor location. In block 209, the component adds the EM data along with the label to the training data and then loops to block 205 to select the next EM output. In block 210, the component trains the classifier using the training data and then completes”; and [0083], “In some embodiments, the MLG system may speed up the generating of derived EM data that is derived from the modeled EM outputs generated for the arrhythmia models of the arrhythmia model libraries. The derived EM data may be a VCG (or other cardiogram) generated from a modeled EM output (e.g., 3,000 EM meshes). The MLG system may group together arrhythmia models with source configurations that have similar anatomical parameters. Each arrhythmia model in a group will thus have similar electrophysiology parameters and different anatomical parameters. The MLG system then runs the simulation for a representative arrhythmia model of the group. The MLG system, however, does not need to run the simulations for the other arrhythmia models in the group. To generate the VCG for one of the other arrhythmia models, the MLG system inputs the modeled EM output of the representative arrhythmia model and the anatomical parameters of the other arrhythmia model. The MLG system then calculates the VCG values for the other arrhythmia model based on the values of the modeled EM output with adjustments based on differences in the anatomical parameters of the representative arrhythmia model and the other source configuration. In this way, the MLG system avoids running any simulations except for the representative arrhythmia model of each group of arrhythmia models”. Note that the electrophysiology parameters and different anatomical parameters are mapped to the anatomical and electrophysiological properties, and the Em outputs are mapped to the training data).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Baram to have the method of claim 4 wherein the neural network comprises a trained neural network that has been trained with a training corpus comprising a plurality of synthetic EAM images that were generated using a variety of different anatomical and electrophysiological properties as taught by Villongco in order to enable interpolating model parameter for a vertex based on parameters of vertices of polyhedrons and adjusting patient electromagnetic data based on difference between the model orientation and the patient orientation (See Villongco: Fig. 25, and [0099], “FIG. 25 is a block diagram that illustrates the process of converting an arrhythmia model based on a first polyhedron to an arrhythmia model based on a second polyhedron in some embodiments. A first polyhedron arrhythmia model of a first polyhedron arrhythmia model 2501 is input to an extract surface component 2502. The extract surface component extracts the surface of the mesh of the arrhythmia model. A populate with second polyhedrons component 2303 inputs the surface and populates the volume within the surface with the second type of polyhedrons. An interpolate model parameters component 2504 inputs the second polyhedron mesh and the first polyhedral model and generates model parameters of the second polyhedral model and outputs the second polyhedral model 2505”). Baram teaches a method and system that may reconstruct the left atrium of the patients based on the sparse location measurements using electro-anatomic mapping techniques with neural neatwork modeling; while Villongco teaches a system and method that may generate and interpolate dynamically anatomical and electrophysiological parameters for modeling and training the patient’s organ (heart) modeling. Therefore, it is obvious to one of ordinary skill in the art to modify Baram by Villongco to refine and adjust the parameters for heart modeling for a particular patient’s heart. The motivation to modify Baram by Villongco is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 6, Baram, Severino, Barve, and Villongco teach all the features with respect to claim 5 as outlined above. Further, Baram teaches that the method of claim 5 wherein the neural network comprises a convolutional neural network (See Baram: Fig. 3, and [0004], “As another example, U.S. Patent Application Publication 2017/0046616 describes a use of 3D deep convolutional neural network architecture (DCNNA) equipped with so-called subnetwork modules which perform dimensionality reduction operations on 3D radiological volume before the 3D radiological volume is subjected to computationally expensive operations. Also, the subnetworks convolve 3D data at multiple scales by subjecting the 3D data to parallel processing by different 3D convolutional layer paths. Such multi-scale operations are computationally cheaper than the traditional CNNs that perform serial convolutions. In addition, performance of the subnetworks is further improved through 3D batch normalization (BN) that normalizes the 3D input fed to the subnetworks, which in turn increases learning rates of the 3D DCNNA. After several layers of 3D convolution and 3D sub-sampling with 3D across a series of subnetwork modules, a feature map with reduced vertical dimensionality is generated from the 3D radiological volume and fed into one or more fully connected layers”).
Regarding claim 15, Baram, Severino, and Barve teach all the features with respect to claim 14 as outlined above. Further, Villongco teaches that the apparatus of claim 14 wherein the neural network comprises a trained neural network that has been trained with a training corpus comprising a plurality of synthetic EAM images that were generated using a variety of different anatomical and electrophysiological properties (See Villongco: Fig. 2, and [0061], “FIG. 2 is a flow diagram that illustrates the overall processing of generating a classifier by the MLMO system in some embodiments. A generate classifier component 200 is executed to generate a classifier. In block 201, the component accesses the computational model to be used to run the simulations. In block 202, the component selects the next source configuration (i.e., parameter set) to be used in a simulation. In decision block 203, if all the source configurations have already been selected, then the component continues at block 205, else the component continues at block 204. In block 204, the component runs the simulation using the selected source configuration to generate an EM output for the simulation and then loops to block 202 to select the next source configuration. In block 205, the component selects the next EM output that was generated by a simulation. In decision block 206, if all the EM outputs have already been selected, then the component continues at block 210, else the component continues at block 207. In block 207, the component derives the EM data from the EM output. For example, the EM output may be a collection of EM meshes, and the EM data may be an ECG or a VCG derived from the electromagnetic values of the EM mesh. In some embodiments, the component may in addition identify cycles (periodic intervals of arrhythmic activity) within the ECG or VCG. A cycle may be delimited by successive crossings from a negative voltage to a positive voltage (“positive crossings”) or successive crossings from a positive voltage to a negative voltage (“negative crossings”) with respect to a spatial direction or set of directions comprising a reference frame or set of reference frames. A reference frame may coincide with anatomical axes (e.g., left-to-right with x, superior-to-inferior with y, anterior-to-posterior with z), imaging axes (e.g., CT, MR, or x-ray coordinate frames), body-surface lead vectors, principal axes computed by principal component analysis of measured or simulated EM source configurations and outputs, or user-defined directions of interest. For example, a three-second VCG may have three cycles, and each cycle may be delimited by the times of the positive crossings along the x-axis. Alternatively, the cycles may be delimited by crossings along the y-axis or z-axis. In addition, cycles may be defined by negative crossings. Thus, in some embodiments, the component may generate training data from a single VCG based on various cycle definitions that are various combinations of positive crossings and negative crossings with the cycles for all the axes being defined by crossings on one of the x-axis, y-axis, and z-axis or the cycles for each defined by crossings on that axis. Moreover, the training data may include cycles identified based on all possible cycle definitions or a subset of the cycle definition. For example, the training data may include, for each axis, a cycle defined by positive crossings of the x-axis, negative crossings of the y-axis, and positive crossings of that axis itself. Cycles definitions may also be defined by the timing of electrical events derived from the values stored in the EM mesh. For example, a point or set of points in the mesh may periodically cross voltage thresholds signifying electrical activation and deactivation. Thus, a cycle may be defined by activation-deactivation, or successive activation-activation or deactivation-deactivation intervals corresponding to a point or set of points within the mesh. The resulting timings of these intervals can be co-localized to the ECG or VCG for cycles identification. In block 208, the component labels the EM data based on the source configuration (e.g., a source location). When cycles are identified, the component may label each cycle with the same label. For example, the component may label the identified cycles with the same rotor location. In block 209, the component adds the EM data along with the label to the training data and then loops to block 205 to select the next EM output. In block 210, the component trains the classifier using the training data and then completes”; and [0083], “In some embodiments, the MLG system may speed up the generating of derived EM data that is derived from the modeled EM outputs generated for the arrhythmia models of the arrhythmia model libraries. The derived EM data may be a VCG (or other cardiogram) generated from a modeled EM output (e.g., 3,000 EM meshes). The MLG system may group together arrhythmia models with source configurations that have similar anatomical parameters. Each arrhythmia model in a group will thus have similar electrophysiology parameters and different anatomical parameters. The MLG system then runs the simulation for a representative arrhythmia model of the group. The MLG system, however, does not need to run the simulations for the other arrhythmia models in the group. To generate the VCG for one of the other arrhythmia models, the MLG system inputs the modeled EM output of the representative arrhythmia model and the anatomical parameters of the other arrhythmia model. The MLG system then calculates the VCG values for the other arrhythmia model based on the values of the modeled EM output with adjustments based on differences in the anatomical parameters of the representative arrhythmia model and the other source configuration. In this way, the MLG system avoids running any simulations except for the representative arrhythmia model of each group of arrhythmia models”. Note that the electrophysiology parameters and different anatomical parameters are mapped to the anatomical and electrophysiological properties, and the Em outputs are mapped to the training data).
Regarding claim 16, Baram, Severino, Barve, and Villongco teach all the features with respect to claim 15 as outlined above. Further, Baram teaches that the apparatus of claim 15 wherein the neural network comprises a convolutional neural network (See Baram: Fig. 3, and [0004], “As another example, U.S. Patent Application Publication 2017/0046616 describes a use of 3D deep convolutional neural network architecture (DCNNA) equipped with so-called subnetwork modules which perform dimensionality reduction operations on 3D radiological volume before the 3D radiological volume is subjected to computationally expensive operations. Also, the subnetworks convolve 3D data at multiple scales by subjecting the 3D data to parallel processing by different 3D convolutional layer paths. Such multi-scale operations are computationally cheaper than the traditional CNNs that perform serial convolutions. In addition, performance of the subnetworks is further improved through 3D batch normalization (BN) that normalizes the 3D input fed to the subnetworks, which in turn increases learning rates of the 3D DCNNA. After several layers of 3D convolution and 3D sub-sampling with 3D across a series of subnetwork modules, a feature map with reduced vertical dimensionality is generated from the 3D radiological volume and fed into one or more fully connected layers”).
Claims 8-9 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Baram, etc. (US 20200029845 A1) in view of Severino (US 20160183824 A1), further in view of Barve, etc. (US 20250245829 A1) and Chen, etc. (US 20210279952 A1).
Regarding claim 8, Baram, Severino, and Barve teach all the features with respect to claim 7 as outlined above. However, Baram, modified by Severino and Barve, fails to explicitly disclose that the method of claim 7 further comprising: processing the texture map and the initial three-dimensional mesh via a differentiable renderer and an optimization routine to provide a more finely-tuned texture map and three-dimensional mesh that more closely match information comprising the EAM two-dimensional image information.
However, Chen teaches that the method of claim 7 further comprising: processing the texture map and the initial three-dimensional mesh via a differentiable renderer and an optimization routine to provide a more finely-tuned texture map and three-dimensional mesh that more closely match information comprising the EAM two-dimensional image information (See Chen: Fig. 2, and [0025], “FIG. 2 illustrates an example training pipeline 200 that can be utilized in accordance with various embodiments. This example pipeline 200 includes two different renderers. The first renderer is a generator network, such as a GAN (e.g., StyleGAN), and the second renderer in this example is a differentiable graphics renderer, such as an interpolation-based differentiable renderer (DIB-R). A DIB-R is a differentiable rendering framework that allows gradients to be analytically computed for all pixels in an image. This framework can view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry, allowing for accurate optimization over vertex positions, colors, normals, light directions, and texture coordinates through a variety of lighting models. In this example, the generator 204 is used as a synthetic data generator with efficient annotation of a multi-view dataset 206. This dataset can then be used to train an inverse graphics network 212 that predicts 3D properties from the 2D images. This network can be used to disentangle the latent code of the generator through a carefully designed mapping network”; and Figs. 13-14, and [0123], “In at least one embodiment, training pipeline 1404 (FIG. 14), a scenario may include facility 1302 requiring a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from model registry 1324 may not be fine-tuned or optimized for imaging data 1308 generated at facility 1302 because of differences in populations, robustness of training data used to train a machine learning model, diversity in anomalies of training data, and/or other issues with training data. In at least one embodiment, AI-assisted annotation 1310 may be used to aid in generating annotations corresponding to imaging data 1308 to be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 may be used as ground truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training 1314. In at least one embodiment, model training 1314—e.g., AI-assisted annotations 1310, labeled clinic data 1312, or a combination thereof—may be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model 1316, and may be used by deployment system 1306, as described herein”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Baram to have the method of claim 7 further comprising: processing the texture map and the initial three-dimensional mesh via a differentiable renderer and an optimization routine to provide a more finely-tuned texture map and three-dimensional mesh that more closely match information comprising the EAM two-dimensional image information as taught by Chen in order to avoid unrealistic repetition or omissions, thus obtaining a large number of three dimensional models in a simple, inexpensive, and time- and resource-efficient manner (See Chen: Fig. 11, and [0003], “A variety of different industries rely upon three-dimensional (3D) modeling for various purposes, including those that require the generation of representations of 3D environments. In order to provide realistic complex environments, it is necessary to have a variety of different types of objects, or similar objects with different appearances, to avoid unrealistic repetition or omissions”). Baram teaches a method and system that may reconstruct the left atrium of the patients based on the sparse location measurements using electro-anatomic mapping techniques with neural neatwork modeling; while Chen teaches a system and method that may reconstruct the 3D object models realistically using the differentiable renderers and optimizing the neural network model. Therefore, it is obvious to one of ordinary skill in the art to modify Baram by Chen to use the differential renderer and neural network optimization to model the 3D object more realistic. The motivation to modify Baram by Chen is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 9, Baram, Severino, Barve, and Chen teach all the features with respect to claim 8 as outlined above. Further, Barve and Chen teach that the method of claim 8 further comprising:
deforming the approximate three-dimensional EAM model to three-dimensional cardiac geometry information to provide a deformed three-dimensional EAM model (See Barve: Fig. 1, and [0069], “With continued reference to FIG. 1, in some cases, once modes of variation are extracted, processor 104 may be configured to create a shape representation for any given heart shape within the studied class. In a non-limiting example, 3D model 156 having a shape S may be mathematically represented as S=S+Σ.sub.k=1.sup.Ma.sub.k×ϕ.sub.k, wherein S denotes the mean shape derived from the set of example shapes, M is the number of modes of variation considered, a.sub.k are the coefficients or weights for each mode, and ϕ.sub.k are the modes of variation (eigenvectors corresponding to the kth principal component). In some cases, coefficients a.sub.k may dictate a degree to which each mode of variation is present in shape S. In some cases, coefficients a.sub.k may vary from positive to negative (or negative to positive) based on the deformation of the 3D model 156 in directions described by each mode of variation. In some cases, 3D model 156 may include mean shape as described herein. In some cases, 3D model 156 may include a predictive heart shape that may not have been explicitly seen in the set of example shapes or patient's heart observations. In some cases, 3D model 156 may be in 3D VOR as described above”; and [0071], “Still referring to FIG. 1, in some embodiments, SSM may be used to refine a mesh produced using a 3D model estimate. For example, a machine learning model such as a deep neural network may be used to produce a voxel grid estimate of patient's organ based on ICE frames of the patient's organ, and SSM may be used to refine this estimate to produce a 3D mesh of patient's organ. In some embodiments, this may include deforming a template of patient's organ to arrive at a most probable shape that matches a DNN estimate. Additionally, or alternatively, a machine learning model may be trained to directly generate parameters of a statistical shape model”);
mapping the more finely-tuned texture map from the deformed three-dimensional EAM model to the three-dimensional cardiac geometry to provide the generated three-dimensional EAM presentation of the particular patient's heart (See Chen: Fig. 3, and [0026], “A mapping network can be trained and used to map the viewpoint, shape (e.g., mesh), texture, and background into the latent code of the generator. Since the generator may not be completely disentangled, the entire generative model can be fine-tuned while keeping the inverse graphics network fixed. A mapping network, such as the example illustrated in FIG. 3, can map the viewpoints to the first four layers and map the shape, texture, and background to the last twelve layers of W*. For simplicity, the first four layers can be denoted as WV*, and the last twelve layers as WS*TB, where WV*∈R2048 and WS*TB∈ R3008. It should be noted that there may be different numbers of layers in other models or networks. In this example, the mapping network gν for viewpoint V and gs for shape S are separate MLPs while gt for texture T and gb for background B are CNN layers”).
Regarding claim 18, Baram, Severino, and Barve teach all the features with respect to claim 17 as outlined above. Further, Chen teaches that the apparatus of claim 17 wherein the control circuit is further configured to: process the texture map and the initial three-dimensional mesh via a differentiable renderer and an optimization routine to provide a more finely-tuned texture map and three-dimensional mesh that more closely match information comprising the EAM two-dimensional image information (See Chen: Fig. 2, and [0025], “FIG. 2 illustrates an example training pipeline 200 that can be utilized in accordance with various embodiments. This example pipeline 200 includes two different renderers. The first renderer is a generator network, such as a GAN (e.g., StyleGAN), and the second renderer in this example is a differentiable graphics renderer, such as an interpolation-based differentiable renderer (DIB-R). A DIB-R is a differentiable rendering framework that allows gradients to be analytically computed for all pixels in an image. This framework can view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry, allowing for accurate optimization over vertex positions, colors, normals, light directions, and texture coordinates through a variety of lighting models. In this example, the generator 204 is used as a synthetic data generator with efficient annotation of a multi-view dataset 206. This dataset can then be used to train an inverse graphics network 212 that predicts 3D properties from the 2D images. This network can be used to disentangle the latent code of the generator through a carefully designed mapping network”; and Figs. 13-14, and [0123], “In at least one embodiment, training pipeline 1404 (FIG. 14), a scenario may include facility 1302 requiring a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from model registry 1324 may not be fine-tuned or optimized for imaging data 1308 generated at facility 1302 because of differences in populations, robustness of training data used to train a machine learning model, diversity in anomalies of training data, and/or other issues with training data. In at least one embodiment, AI-assisted annotation 1310 may be used to aid in generating annotations corresponding to imaging data 1308 to be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 may be used as ground truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training 1314. In at least one embodiment, model training 1314—e.g., AI-assisted annotations 1310, labeled clinic data 1312, or a combination thereof—may be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model 1316, and may be used by deployment system 1306, as described herein”).
Regarding claim 19 Baram, Severino, Barve, and Chen teach all the features with respect to claim 18 as outlined above. Further, Barve and Chen teach that the apparatus of claim 18 wherein the control circuit is further configured to: deform the approximate three-dimensional EAM model to three-dimensional cardiac geometry information to provide a deformed three-dimensional EAM model (See Barve: Fig. 1, and [0069], “With continued reference to FIG. 1, in some cases, once modes of variation are extracted, processor 104 may be configured to create a shape representation for any given heart shape within the studied class. In a non-limiting example, 3D model 156 having a shape S may be mathematically represented as S=S+Σ.sub.k=1.sup.Ma.sub.k×ϕ.sub.k, wherein S denotes the mean shape derived from the set of example shapes, M is the number of modes of variation considered, a.sub.k are the coefficients or weights for each mode, and ϕ.sub.k are the modes of variation (eigenvectors corresponding to the kth principal component). In some cases, coefficients a.sub.k may dictate a degree to which each mode of variation is present in shape S. In some cases, coefficients a.sub.k may vary from positive to negative (or negative to positive) based on the deformation of the 3D model 156 in directions described by each mode of variation. In some cases, 3D model 156 may include mean shape as described herein. In some cases, 3D model 156 may include a predictive heart shape that may not have been explicitly seen in the set of example shapes or patient's heart observations. In some cases, 3D model 156 may be in 3D VOR as described above”; and [0071], “Still referring to FIG. 1, in some embodiments, SSM may be used to refine a mesh produced using a 3D model estimate. For example, a machine learning model such as a deep neural network may be used to produce a voxel grid estimate of patient's organ based on ICE frames of the patient's organ, and SSM may be used to refine this estimate to produce a 3D mesh of patient's organ. In some embodiments, this may include deforming a template of patient's organ to arrive at a most probable shape that matches a DNN estimate. Additionally, or alternatively, a machine learning model may be trained to directly generate parameters of a statistical shape model”);
map the more finely-tuned texture map from the deformed three-dimensional EAM model to the three-dimensional cardiac geometry to provide the generated three-dimensional EAM presentation of the particular patient's heart (See Chen: Fig. 3, and [0026], “A mapping network can be trained and used to map the viewpoint, shape (e.g., mesh), texture, and background into the latent code of the generator. Since the generator may not be completely disentangled, the entire generative model can be fine-tuned while keeping the inverse graphics network fixed. A mapping network, such as the example illustrated in FIG. 3, can map the viewpoints to the first four layers and map the shape, texture, and background to the last twelve layers of W*. For simplicity, the first four layers can be denoted as WV*, and the last twelve layers as WS*TB, where WV*∈R2048 and WS*TB∈ R3008. It should be noted that there may be different numbers of layers in other models or networks. In this example, the mapping network gν for viewpoint V and gs for shape S are separate MLPs while gt for texture T and gb for background B are CNN layers”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights