DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Obligation Under 37 CFR 1.56 – Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Response to Amendment
Applicant’s amendment filed on February 24, 2026 has been entered. Claims 1-11 have been amended, and new claims 12-20 have been added. Thus, claims 1-20 are pending in this application, with claims 1, 4, 7 and 10 being independent.
Applicant’s amendment of February 24, 2026 overcomes the following objections/rejections:
Objection to claim 4.
Rejection of claim 6 under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art;
Ascertaining the differences between the prior art and the claims at issue;
Resolving the level of ordinary skill in the pertinent art; and
Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over MUSSACK et al. (US 2007/0046966, hereinafter “MUSSACK”) in view of VERONESI et al. (US 2018/0344290, hereinafter “VERONESI”).
Regarding claim 7, MUSSACK discloses a visual data delivery system (FIGS. 1-2: Server 102 and/or FIG. 3: Server 302; ¶ [0018]: “a system for medical image processing comprising: a server comprising server processing resources and capable of storing three dimensional image data;” ¶ [0030]: “FIG. 1 shows a block diagram of a distributed network 100 according to an embodiment of the present application. Distributed network 100 may include a server 102, a client 104, and a communication pathway 106.” ) comprising a processing unit (FIG. 2: CPU 202; ¶ [0031]: “Server 102 may include CPU 202, cache memory 204, storage memory 206, RAM 208, operating system 230, and other drivers and applications 232. CPU 202 may include multiple processors.”) and interface circuitry (¶ [0030]: “communication pathway 106.” ¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: The server 102 communicates with communication pathway 106. Thus, the server, by necessity, must include interface circuitry that connects the server with the communication pathway.), the processing unit configured to:
process a sequence of 3-dimensional (3D) images (e.g., ¶ [0072]: “an examination may include 3D cine image data,”) of a body (e.g., ¶ [0072]: “An examination may be one or more sets of volumetric or two dimensional image data that correspond to a radiological examination of a patient.”) to generate first 2-dimensional (2D) image data (¶ [0012]: “processing 3D image data into 2D image data.”) representing a first sequence of 2D images of the body (e.g., ¶ [0072]: “an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like.” ¶ [0012]: “multi-planar reformatting (MPR),” ¶ [0012]: “In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system.”) (¶ [0072]: “FIG. 8 shows a flowchart of a method 800 illustrating iterative user interaction with an examination displayed on a client according to an embodiment of the present invention. At step 802 a user at a client (similar to client 312 shown in FIG. 3) invokes an application that may be capable of displaying images that correspond to an examination. An examination may be one or more sets of volumetric or two dimensional image data that correspond to a radiological examination of a patient. For example, an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like. The application may be able to display the entire examination, or a portion of the examination. The application may allow the user to interact through a user interface with a displayed portion of the examination, or the entire examination. For example, the application may allow the user to perform the following interactions: page the image up and/or down; pan the image; zoom in and/or out of the image; rotate the image; adjust contrast; adjust brightness, adjust color parameters; adjust grayscale parameters; adjust a slice thickness; change between maximum intensity projection, average intensity projection, and minimum intensity projection; change to volume rendering mode; adjust an angle of viewing. When a user invokes an application for interacting with an examination, it may be desirable to start a corresponding server-based application that can perform and/or assist with image processing. So, for example, the invocation of the client-based application may cause a message to be sent to a server. The message may instruct or request the server to start up an image processing application, if such an application is not already running on the server. For example, the server-based application may be capable of performing 3D processing, such as MIP, MPR, VR and/or the like.” ¶ [0011]: “Thus, to display a medical image on a display, 3D data may be processed to form 2D data.” ¶ [0012]: “A variety of techniques are known for processing 3D image data into 2D image data. These techniques include multi-planar reformatting (MPR), maximum (or minimum) intensity projection (MIP), and volume rendering (VR). In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system. For example, an application incorporating MPR functionality may allow a user to rotate a displayed image at any angle and centered at any location within the volume. Thus, MPR allows a clinician to view the anatomy from any of a variety of positions and angles.” ¶ [0014]: “VR processing is another way to display a 3D volume in 2D. VR processing renders the surface and/or interior of an object, making the surface of the object appear solid, transparent and/or translucent. Objects inside the interior of a volume of interest (such as organs, blood vessels, bones, etc.) may also be made to appear solid, transparent, and/or translucent.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0031]: “For example, other drivers and applications 232 may include a graphics accelerator hardware module and a three-dimensional graphics processor software module. Other drivers and applications 232 may include drivers and applications intended to facilitate medical image processing. As used in this application, image processing is a broad term, including, for example, image display and volume rendering. Image processing may include, for example, processing of 2D, 3D, and 2D-projection image data, and the like. Image processing may include, for example, MPR, MIP, VR, and the like. Image data may be formatted with a variety of formats, including, for example, DICOM, ANALYZE, BMP, JPG, GIF, TIF, and the like.” ¶ [0042]: “After step 702, the first branch 740 continues to step 704, in which 3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data.” ¶ [0043]: “The second branch 750 proceeds after step 702 to step 712, in which a server processes 3D image data to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. ),
wherein the 2D images are images of the body (¶ [0012]: “the anatomy”) in a 2D image plane (¶ [0012]: “a 2D slice”) through the 3D images (¶ [0012]: “3D volume”) ( e.g., ¶ [0072]: “an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like.” ¶ [0012]: “multi-planar reformatting (MPR),” ¶ [0012]: “In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system.” ¶ [0012]: “A variety of techniques are known for processing 3D image data into 2D image data. These techniques include multi-planar reformatting (MPR), maximum (or minimum) intensity projection (MIP), and volume rendering (VR). In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system. For example, an application incorporating MPR functionality may allow a user to rotate a displayed image at any angle and centered at any location within the volume. Thus, MPR allows a clinician to view the anatomy from any of a variety of positions and angles.”), and
wherein an amount of data representing the first 2D image data is less than an amount of data representing the 3D images from which the first 2D image data is generated (¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume.”);
send, via the interface circuitry (FIG. 2: “communication pathway 106”; and/or FIG. 3: “communication pathway 305”), the first 2D image data to a display system (FIG. 2: Client 104 including Display Driver 212 and Display 214; and/or FIG. 3: Client 312) remote from the visual data delivery system (e.g., as shown in FIG. 2, Client 104 is remote from Server 102; and/or, as shown in FIG. 3, Client 312 is remote from Server 302.) (¶ [0032]: “Client 104 may be a personal computer, desktop, laptop, workstation, dumb terminal, thin client, or the like. Client 104 may include CPU 210, display driver 212, display 214, user interface 216, cache memory 218, storage memory 220, video memory 222, RAM 224, operating system 226, and other drivers and applications 228.” ¶ [0040]: “Bandwidth of communication pathway 305 may factor in determining how system resources are allocated in order to display medical images on client 312. Bandwidth may be estimated or measured by either of client 312 or server 302. For example, client monitor 314 or server monitor 304 may estimate bandwidth by any of a number of known techniques. One technique for estimating bandwidth is to communicate a test packet of known size across communication pathway 305, and measure a time for completion of test packet communication.” ¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “a server processes 3D image data to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.”) for display of the first sequence of 2D images of the body by the display system, without sending the 3D images (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.” ¶ [0046]: “At step 404, a monitor monitors bandwidth and system resources.” ¶ [0048]: “At step 408, an allocation of system resources are recommended based on monitor data.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.”);
process the sequence of 3D images to generate one or more 3D models representing the body or a part of the body from the sequence of 3D images (¶ [0014]: “VR processing is another way to display a 3D volume in 2D. VR processing renders the surface and/or interior of an object, making the surface of the object appear solid, transparent and/or translucent. Objects inside the interior of a volume of interest (such as organs, blood vessels, bones, etc.) may also be made to appear solid, transparent, and/or translucent.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.”); and
send, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), the one or more 3D models to the display system (¶ [0014]: “VR processing is another way to display a 3D volume in 2D. VR processing renders the surface and/or interior of an object, making the surface of the object appear solid, transparent and/or translucent. Objects inside the interior of a volume of interest (such as organs, blood vessels, bones, etc.) may also be made to appear solid, transparent, and/or translucent.”) (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.”); or
identify a type of body or type of body part shown in the sequence of 3D images; and send, via the interface circuitry, an indication of the identified type of body or type of body part, or an indication of one or more predetermined 3D models corresponding to the identified type of body or type of body part, to the display system (NOTE: The alternative to this limitation has been met and, as such, this limitation does not need to be given any patentable weight.);
receive, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), a 2D image plane adjustment indication from the display system (e.g., ¶ [0082]: “the user opens a control panel in the software application, and enters a new viewing angle”) (¶ [0047]: “At step 406, there is a request to display an image on client. For example, client 312 may be running image processing software. A user may interact with the software to perform an image processing task on a medical image. The software may initiate an image processing request that is relayed to the operating system of the client 312. This image processing request may include information about the image to be processed, such as image quality. The image processing request may also include information about the nature of the image processing request, such as to zoom or pan the image, for example. The image processing request may be communicated from the client 312 to the server 302.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”),
wherein the 2D image plane adjustment indication indicates a required rotation and/or translation of the 2D image plane (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”);
process the sequence of 3D images and/or a further sequence of 3D images to generate second 2D image data representing a second sequence of 2D images of the body (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”),
wherein the 2D images in the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane (¶ [0012]: “A variety of techniques are known for processing 3D image data into 2D image data. These techniques include multi-planar reformatting (MPR), maximum (or minimum) intensity projection (MIP), and volume rendering (VR). In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system. For example, an application incorporating MPR functionality may allow a user to rotate a displayed image at any angle and centered at any location within the volume. Thus, MPR allows a clinician to view the anatomy from any of a variety of positions and angles.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”); and
send, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), the second 2D image data to the display system for display of the second sequence of 2D images of the body by the display system (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”).
Whereas MUSSACK may not be entirely explicit as to, VERONESI discloses a visual data delivery system (e.g., FIG. 1) comprising a processing unit (e.g., “Processor” 116 in FIG. 1) and interface circuitry (¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.” ¶ [0019]: “A video processor module may be provided that reads the image volumes from a memory and displays an image in real time while a procedure is being carried out on a patient. A video processor module may store the images in the memory 120, from which the images are read and displayed.”), the processing unit configured to:
process a sequence of 3-dimensional (3D) images of a body (¶ [0021]: “The ultrasound data may comprise a 3D ultrasound dataset or in some examples a 4D ultrasound dataset. For example, the ultrasound data may comprise a volume of data including 3D color Doppler data over time, such as over one or more heart cycles (e.g., ultrasound echocardiography data), and may be stored in a memory device.” ¶ [0039]: “It should be appreciated that although the method 200 is described with regard to 3D ultrasound datasets, method 200 may be applied to four-dimensional datasets comprising three spatial dimensions plus time.” ¶ [0017]: “A memory 120 is included for storing processed volumes of acquired data. In an exemplary embodiment, the memory 120 is of sufficient capacity to store at least several seconds worth of volumes of ultrasound data. The volumes of data are stored in a manner to facilitate retrieval thereof according to its order or time of acquisition.” ¶ [0021]: “a region of interest (ROI).” ¶ [0024]: “a particular region or organ of interest.” NOTE: A 4D data set comprising 3D image data over time is a sequence of 3D image data.) to generate first 2-dimensional (2D) image data representing a first sequence of 2D images of the body (e.g., ¶ [0020]: “intersection information” ¶ [0031]: “generating one or more 2D slice image(s) from the 3D ultrasound dataset”) (¶ [0020]: “FIG. 2 shows a high-level flow chart illustrating an example method 200 for displaying intersection information on an ultrasound image according to an embodiment. Method 200 will be described herein with reference to the system and components depicted in FIG. 1, though it should be understood that the method may be applied to other systems and components without departing from the scope of the present disclosure. Method 200 may be carried out by processor 116, and may be stored as executable instructions in non-transitory memory of the processor 116.” ¶ [0021]: “Method 200 begins at 205. At 205, method 200 includes acquiring ultrasound data for a region of interest (ROI). In some examples, the ROI may comprise a structure or object, for example an organ such as a human heart or a region thereof. The ultrasound data may comprise a 3D ultrasound dataset or in some examples a 4D ultrasound dataset. For example, the ultrasound data may comprise a volume of data including 3D color Doppler data over time, such as over one or more heart cycles (e.g., ultrasound echocardiography data), and may be stored in a memory device.” ¶ [0024]: “At 215, method 200 includes receiving an indication of the ROI. The indication of the ROI may comprise an indication received, from a user via a user interface such as user interface 115, of a particular region or organ of interest. In some examples, the indication may include two or more ROIs. For example, the user may desire to image the heart and more specifically the mitral valve of the heart, and therefore may indicate that the ROI includes both the heart and the mitral valve.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0030]: “As another example, one or more planes may be automatically identified and indicated based on the indication of the ROI. For example, standard views may be predetermined for a given ROI, and the indication of the slice positions corresponding to the standard views may be automatically retrieved and/or generated based on the indication of the ROI as well as the 3D mesh model. In other words, the slice positions or planes may be located at fixed pre-determined positions relative to the data volume or the ultrasound probe. For example, two orthogonal slice planes corresponding to the azimuth and elevation planes of the acquired ultrasound ROI may be positioned such that the planes intersect the center of the data volume. As another example, three slice planes may be rotated about a common axis (such as the probe axis) where the planes are by default oriented to provide visualization of a four chamber view, a two chamber view, and a long axis view of the left ventricle of the heart. The user may or may not modify the position and orientation of these planes.” ¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.” ¶ [0043]: “The first 2D slice 330 and the second 2D slice 350 may comprise different 2D views of the 3D ultrasound dataset.” NOTE: The 4D dataset clearly comprises a sequence of 3D image data, and, as such, the 2D slice images of the 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).),
wherein the 2D images are images of the body in a 2D image plane through the 3D images (¶ [0002]: “to visualize intersections between volume data and planes.” ¶ [0002]: “display one or more 2D slice planes reconstructed from a 3D ultrasound data volume,” ¶ [0011]: “planes within the 3D dataset for rendering.” ¶ [0029]: “a determination is made as to the coordinates of the plane(s) through the 3D volume dataset” ¶ [0029]: “the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.” ¶ [0043]: “The first 2D slice 330 and the second 2D slice 350 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 330 and 350 with the 3D image volume 305 may be visually depicted in the 3D mesh model 320. For example, the plane corresponding to the first 2D slice 330 may be represented as a line or intersection 335 on the 3D mesh model 320, while the plane corresponding to the second 2D slice 350 may be represented as a line or intersection 355 on the 3D mesh model 320. The intersections 335 and 355 may also be displayed or overlaid on the 3D image volume 305.” ¶ [0051]: “The first 2D slice 430, the second 2D slice 450, and the third 2D slice 470 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 430, 450, and 470 with the 3D image volume 405 may be visually depicted in the 3D mesh model 420.”), and
wherein an amount of data representing the first 2D image data is less than an amount of data representing the 3D images from which the first 2D image data is generated (NOTE: The amount of data representing the first 2D image data must be less than the amount of data representing the 3D images from which the first 2D image data is generated since the 2D image data is a subset (i.e., a slice) of the 3D image data. Thus, this limitation is inherently taught by VERONESI.);
send, via the interface circuitry (e.g., ¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.” ¶ [0019]: “A video processor module may be provided that reads the image volumes from a memory and displays an image in real time while a procedure is being carried out on a patient. A video processor module may store the images in the memory 120, from which the images are read and displayed.”), the first 2D image data to a display system (e.g., FIG. 1: “Display device” 118; ¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.”) for display of the first sequence of 2D images of the body by the display system (¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.”),
process the sequence of 3D images to
generate one or more 3D models representing the body or a part of the body from the sequence of 3D images (¶ [0023]: “At 210, method 200 includes rendering a 3D image volume from the ultrasound data. For example, a 3D image volume or 3D image may be reconstructed from the 3D ultrasound dataset using any suitable 3D volumetric image reconstruction technique. In some embodiments, the 3D volume dataset is displayed in real-time, for example, on the display device 118.” ¶ [0027]: “At 225, method 200 includes generating a 3D mesh model of the ROI. The 3D mesh model of the ROI may be retrieved from a digital repository of 3D mesh models corresponding to different organs or anatomies, as an illustrative and non-limiting example. As another example, the method may generate a 3D mesh model of the segmented ROI obtained at 220. In yet another example, the method may retrieve a generic 3D mesh model of the ROI from a digital repository and fit the mesh model to the segmented ROI obtained at 220.” ¶ [0028]: “Method 200 further includes coupling the 3D mesh model to the 3D image. For example, one or more points of the 3D mesh model may be linked to one or more corresponding voxels of the ROI in the 3D image. In this way, the scale and orientation of the 3D mesh model may correspond to the scale and orientation of the ROI in the 3D image. The coupling may be carried out automatically or in some examples may be carried out with assistance from user input. For example, the method may automatically identify corresponding points in the 3D mesh model and the 3D image, or a user may manually identify, via the user interface 115 for example, the corresponding points in the 3D mesh model and the 3D image.”); and
send, via the interface circuitry (e.g., ¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.”), the one or more 3D models to the display system (¶ [0023]: “At 210, method 200 includes rendering a 3D image volume from the ultrasound data. For example, a 3D image volume or 3D image may be reconstructed from the 3D ultrasound dataset using any suitable 3D volumetric image reconstruction technique. In some embodiments, the 3D volume dataset is displayed in real-time, for example, on the display device 118.” ¶ [0042]: “As depicted, the 3D mesh model 320 of the ROI 308 may be displayed adjacent to the 3D image volume 305 in the graphical user interface 300.”); or
identify a type of body or type of body part shown in the sequence of 3D images; and send, via the interface circuitry, an indication of the identified type of body or type of body part, or an indication of one or more predetermined 3D models corresponding to the identified type of body or type of body part, to the display system (NOTE: The alternative to this limitation has been met and, as such, this limitation is not required to be given patentable weight.);
receive, via the interface circuitry (¶ [0014]: “A user interface 115 may be used to control operation of the ultrasound imaging system 100, including controlling the input of patient data, changing a scanning or display parameter, and the like. The user interface 115 may include a graphical user interface configured for display on a display device 118. The graphical user interface may include information to be output to a user (such as ultrasound images, patient data, etc.) and may also include menus or other elements through which a user may enter input to the computing system. In examples described in more detail below with respect to FIGS. 2-4, the user interface may receive inputs from a user indicating, for example, adjustments to the position of planes to be imaged. The user interface 115 may include one or more of the following: a rotary, a mouse, a keyboard, a trackball, a touch-sensitive display, hard keys linked to specific actions, soft keys that may be configured to control different functions, and a graphical user interface.”), a 2D image plane adjustment indication from the display system (¶ [0014]: “with respect to FIGS. 2-4, the user interface may receive inputs from a user indicating, for example, adjustments to the position of planes to be imaged.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0036]: “At 255, method 200 includes determining if one or more of the 2D slice(s) will be adjusted. Determining if one or more of the 2D slices will be adjusted comprises determining if user input regarding a new position for a 2D slice on the 3D mesh model and/or the 3D image volume. For example, the user may select an intersection displayed on the 3D mesh model and manually move or adjust the position of the intersection within the 3D mesh model, for example via the user interface 115. In some examples, the user may adjust the display of the 3D mesh model, for example by scaling the display of the 3D mesh model and/or rotating or otherwise changing the displayed perspective of the 3D mesh model. Changing the display of the 3D mesh model makes it easier for the user to navigate the 3D volume to select a new position for a 2D slice.” ¶ [0037] If there is no user input relating to a new position for a 2D slice, then there are no slice adjustments (“NO”), and method 200 returns. However, if there is user input relating to a new position for a 2D slice, there is a slice adjustment (“YES”), and method 200 continues to 260. At 260, method 200 includes receiving an indication of a new slice position, for example via the user interface 115.” ¶ [0038]: “At 265, method 200 includes generating and displaying a new 2D slice from the ultrasound dataset at the indicated position. At 270, method 200 includes displaying the updated intersections between the new 2D slice, the 3D mesh model, and the 3D image. Method 200 then returns.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ),
wherein the 2D image plane adjustment indication indicates a required rotation and/or translation of the 2D image plane (¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image through the 3D model and/or 3D image dataset. Thus, this limitation is inherently taught by VERONESI.);
process the sequence of 3D images and/or a further sequence of 3D images to generate second 2D image data representing a second sequence of 2D images of the body (¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” ¶ [0065] In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: As already noted herein-above, a 4D dataset clearly comprises a sequence of 3D image data, and, as such, the 2D slice images of a 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).),
wherein the 2D images in the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane (¶ [0060]: “receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image relative to the 3D model and/or corresponding 3D dataset. Thus, this limitation is inherently taught by VERONESI.); and
send, via the interface circuitry (e.g., ¶ [0062]: “system comprises a display device and a processor communicatively coupled to the display device,”), the second 2D image data to the display system for display of the second sequence of 2D images of the body by the display system (¶ [0038] At 265, method 200 includes generating and displaying a new 2D slice from the ultrasound dataset at the indicated position. At 270, method 200 includes displaying the updated intersections between the new 2D slice, the 3D mesh model, and the 3D image. Method 200 then returns.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0065] In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: As already noted herein-above, the 2D slice images of a 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).).
Thus, in order to obtain a distributed medical image processing system having the cumulative features and/or functionalities taught by MUSSACK and VERONESI, it would have been obvious to one of ordinary skill in the art to have modified the distributed medical image processing system taught by MUSSACK so as to incorporate image processing including: processing a sequence of 3-dimensional (3D) images of a body to generate first 2-dimensional (2D) image data representing a first sequence of 2D images of the body; processing the sequence of 3D images to generate one or more 3D models representing the body or a part of the body from the sequence of 3D images; sending, via the interface circuitry, the one or more 3D models to the display system; receiving, via the interface circuitry, a 2D image plane adjustment indication from the display system; processing the sequence of 3D images and/or a further sequence of 3D images to generate second 2D image data representing a second sequence of 2D images of the body, wherein the 2D images in the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane; and sending, via the interface circuitry, the second 2D image data to the display system for display of the second sequence of 2D images of the body by the display system, as taught by VERONESI.
Regarding claim 8 (depends on claim 7), the combination of MUSSACK and VERONESI renders moot:
wherein the identified type of body or identified type of body part is associated with a predetermined 3D model corresponding to the identified type of body or identified type of body part stored at the display system (NOTE: This limitation further limits the alternative limitation in independent claim 7 that has not been given patentable weight. Thus, this limitation is also not required to be given patentable weight since the alternative limitation has been met. ¶ [0027]: “At 225, method 200 includes generating a 3D mesh model of the ROI. The 3D mesh model of the ROI may be retrieved from a digital repository of 3D mesh models corresponding to different organs or anatomies, as an illustrative and non-limiting example. As another example, the method may generate a 3D mesh model of the segmented ROI obtained at 220. In yet another example, the method may retrieve a generic 3D mesh model of the ROI from a digital repository and fit the mesh model to the segmented ROI obtained at 220.” ).
Regarding claim 9 (depends on claim 7), VERONESI further teaches:
wherein the one or more 3D models comprises an indication of a current orientation and/or current position of the 2D image plane with respect to the body (¶ [0032]: “At 240, method 200 includes calculating intersection(s) of the 2D slice(s), the 3D image volume, and the 3D mesh model, which may be easily carried out due to the coupling of the 3D mesh model to the 3D image volume. Calculating the intersections of the 2D slices and the 3D mesh model comprises calculating the location of the planes in the 3D mesh model that correspond to the 2D slices, as well as calculating the location in the 2D slices that intersect with the 3D mesh model.” ¶ [0033]: “At 245, method 200 includes displaying the intersection(s) on the 3D mesh model and the 3D image. For example, lines or curves corresponding to the position of the 2D slices in the 3D mesh model may be displayed or visually overlaid on the 3D mesh model to indicate the intersections of the 2D slices with the 3D mesh model. Similarly, the intersections of the 2D slices with the 3D mesh model may also be displayed or visually overlaid on the 3D image. Alternatively, pixels or voxels of the 3D image corresponding to the intersection of the 2D slices with the 3D image may be colorized or otherwise adjusted to visually indicate the intersections in the 3D image.” ¶ [0034]: “The intersections may be colorized to distinguish the intersections from each other, as well as to provide a visual indication of which 2D slice the intersection corresponds to. For example, one or more visual indicators may be displayed or visually overlaid on the 2D slice; the one or more visual indicators may be colorized similar to the corresponding intersection curve displayed on the 3D mesh model to provide a visual correspondence between the intersection curve and the 2D slice. It should be appreciated that in addition to or as an alternative to colorizing the intersection curves, any parameter may be adjusted to identify or highlight the intersection along the surface. For example, the color, transparency, intensity, and/or value of the pixels corresponding to the identified intersection voxels may be changed.” ¶ [0035]: “At 250, method 200 includes displaying intersection(s) of the 3D model with the 2D slices on the 2D slices. For example, one or more intersection curves may be displayed or visually overlaid on the 2D slices to indicate the intersection of the 2D slice with the 3D mesh model. Alternatively, pixels of the 2D slice corresponding to the location of the intersection(s) may be colorized or otherwise adjusted to indicate the location of the intersection on the 2D slice.” ¶ [0043]: “The first 2D slice 330 and the second 2D slice 350 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 330 and 350 with the 3D image volume 305 may be visually depicted in the 3D mesh model 320. For example, the plane corresponding to the first 2D slice 330 may be represented as a line or intersection 335 on the 3D mesh model 320, while the plane corresponding to the second 2D slice 350 may be represented as a line or intersection 355 on the 3D mesh model 320. The intersections 335 and 355 may also be displayed or overlaid on the 3D image volume 305.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0051]: “The first 2D slice 430, the second 2D slice 450, and the third 2D slice 470 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 430, 450, and 470 with the 3D image volume 405 may be visually depicted in the 3D mesh model 420. For example, the plane corresponding to the first 2D slice 430 may be represented as a line or intersection 435 on the 3D mesh model 420, the plane corresponding to the second 2D slice 450 may be represented as a line or intersection 455 on the 3D mesh model 420, and the plane corresponding to the third 2D slice 470 may be represented as a line or intersection 475 on the 3D mesh model 420. The intersections 435, 455, and 475 may also be displayed or overlaid on the 3D image volume 405 to indicate the positional relationship between the images and image volumes.”), and
wherein the received 2D image plane adjustment indication is a rotation and/or translation of the indication of the 2D image plane in the one or more 3D models, or a rotation and/or translation of the one or more 3D models (¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image relative to the 3D model and/or corresponding 3D dataset. Thus, this limitation is inherently taught by VERONESI.).
Regarding claim 10, MUSSACK discloses a display system (e.g., FIGS. 1-2: Client 104 including Display Driver 212 and Display 214; and/or FIG. 3: Client 312), the display system comprising a processing unit (e.g., FIG. 2: Client CPU 210), a display screen (e.g., FIG. 2: Client Display 214) and interface circuitry (FIG. 1-2: communication pathway 106 and/or FIG. 3: communication pathway 305. ¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: The client 104 communicates through communication pathway 106. Thus, the client, by necessity, must include interface circuitry that connects the client with the communication pathway.), wherein the processing unit is configured to:
receive, via the interface circuitry (FIG. 2: “communication pathway 106”; and/or FIG. 3: “communication pathway 305”), first 2-dimensional (2D) image data (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.”) representing a first sequence of 2D images (e.g., ¶ [0072]: “an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like.” ¶ [0012]: “multi-planar reformatting (MPR),” ¶ [0012]: “In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system.”) of a body (e.g., ¶ [0072]: “An examination may be one or more sets of volumetric or two dimensional image data that correspond to a radiological examination of a patient.” ¶ [0012]: “the anatomy”) from a visual data delivery system remote from the display system (FIGS. 1-2: Server 102 and/or FIG. 3: Server 302; ¶ [0018]: “a system for medical image processing comprising: a server comprising server processing resources and capable of storing three dimensional image data;” ¶ [0030]: “FIG. 1 shows a block diagram of a distributed network 100 according to an embodiment of the present application. Distributed network 100 may include a server 102, a client 104, and a communication pathway 106.” NOTE: As is readily apparent in FIGS. 1-3, Client (104 and/or 312) is remote from Server (102 and/or 302). ) ( ),
wherein the 2D images (¶ [0012]: “processing 3D image data into 2D image data.” ¶ [0072]: “an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like.”) are generated from a sequence of 3-dimensional (3D) images of the body (e.g., ¶ [0072]: “3D cine image data,” ¶ [0012]: “3D volume”) in a 2D image plane (e.g., ¶ [0012]: “a 2D slice”; ¶ [0072]: “an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like.”) (¶ [0012]: “multi-planar reformatting (MPR),” ¶ [0012]: “In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system.” ¶ [0072]: “FIG. 8 shows a flowchart of a method 800 illustrating iterative user interaction with an examination displayed on a client according to an embodiment of the present invention. At step 802 a user at a client (similar to client 312 shown in FIG. 3) invokes an application that may be capable of displaying images that correspond to an examination. An examination may be one or more sets of volumetric or two dimensional image data that correspond to a radiological examination of a patient. For example, an examination may include 3D cine image data, 2D cine image data, 3D static data, 2D static data, and/or the like. The application may be able to display the entire examination, or a portion of the examination. The application may allow the user to interact through a user interface with a displayed portion of the examination, or the entire examination. For example, the application may allow the user to perform the following interactions: page the image up and/or down; pan the image; zoom in and/or out of the image; rotate the image; adjust contrast; adjust brightness, adjust color parameters; adjust grayscale parameters; adjust a slice thickness; change between maximum intensity projection, average intensity projection, and minimum intensity projection; change to volume rendering mode; adjust an angle of viewing. When a user invokes an application for interacting with an examination, it may be desirable to start a corresponding server-based application that can perform and/or assist with image processing. So, for example, the invocation of the client-based application may cause a message to be sent to a server. The message may instruct or request the server to start up an image processing application, if such an application is not already running on the server. For example, the server-based application may be capable of performing 3D processing, such as MIP, MPR, VR and/or the like.” ¶ [0011]: “Thus, to display a medical image on a display, 3D data may be processed to form 2D data.” ¶ [0012]: “A variety of techniques are known for processing 3D image data into 2D image data. These techniques include multi-planar reformatting (MPR), maximum (or minimum) intensity projection (MIP), and volume rendering (VR). In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system. For example, an application incorporating MPR functionality may allow a user to rotate a displayed image at any angle and centered at any location within the volume. Thus, MPR allows a clinician to view the anatomy from any of a variety of positions and angles.” ¶ [0014]: “VR processing is another way to display a 3D volume in 2D. VR processing renders the surface and/or interior of an object, making the surface of the object appear solid, transparent and/or translucent. Objects inside the interior of a volume of interest (such as organs, blood vessels, bones, etc.) may also be made to appear solid, transparent, and/or translucent.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0031]: “For example, other drivers and applications 232 may include a graphics accelerator hardware module and a three-dimensional graphics processor software module. Other drivers and applications 232 may include drivers and applications intended to facilitate medical image processing. As used in this application, image processing is a broad term, including, for example, image display and volume rendering. Image processing may include, for example, processing of 2D, 3D, and 2D-projection image data, and the like. Image processing may include, for example, MPR, MIP, VR, and the like. Image data may be formatted with a variety of formats, including, for example, DICOM, ANALYZE, BMP, JPG, GIF, TIF, and the like.” ¶ [0042]: “After step 702, the first branch 740 continues to step 704, in which 3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data.” ¶ [0043]: “The second branch 750 proceeds after step 702 to step 712, in which a server processes 3D image data to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data.”),
wherein the first 2D image data are received without receiving the 3D images of the body (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.” NOTE: The server process 3D image data to form 2D image data to send to the client. The 3D image data is not sent by the server and, as such, is not received by the client.);
display, using the display screen (¶ [0043]: “the client may process and display 2D image data on a display.”), the first sequence of 2D images (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.” ¶ [0046]: “At step 404, a monitor monitors bandwidth and system resources.” ¶ [0048]: “At step 408, an allocation of system resources are recommended based on monitor data.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.”);
receive, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), one or more 3D models to the display system (¶ [0014]: “VR processing is another way to display a 3D volume in 2D. VR processing renders the surface and/or interior of an object, making the surface of the object appear solid, transparent and/or translucent. Objects inside the interior of a volume of interest (such as organs, blood vessels, bones, etc.) may also be made to appear solid, transparent, and/or translucent.”), and display, using the display screen, the one or more 3D models for the body or a part of the body shown in the first sequence of 2D images (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710. The 2D data may be processed and displayed by the client at step 728.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716. After receipt of 2D image data, the client may further process the 2D image data to form image data for display. At step 728, the client may process and display 2D image data on a display.”); or
receive, via the interface circuitry, an indication of an identified type of body or type of body part, or an indication of one or more predetermined 3D models corresponding to the identified type of body or type of body part, to the display system, and display, using the display screen, one or more predetermined 3D models corresponding to of the identified type of body or type of body part, or an indication of one or more predetermined models corresponding to the identified type of body or type of body part (NOTE: The alternative to this limitation has been met and, as such, this limitation does not need to be given any patentable weight.);
receive, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), a user input from a user of the display system (e.g., ¶ [0082]: “the user opens a control panel in the software application, and enters a new viewing angle”) (¶ [0047]: “At step 406, there is a request to display an image on client. For example, client 312 may be running image processing software. A user may interact with the software to perform an image processing task on a medical image. The software may initiate an image processing request that is relayed to the operating system of the client 312. This image processing request may include information about the image to be processed, such as image quality. The image processing request may also include information about the nature of the image processing request, such as to zoom or pan the image, for example. The image processing request may be communicated from the client 312 to the server 302.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”),
wherein the user input indicates a required rotation and/or translation of the 2D image plane (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”);
send, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), a 2D image plane adjustment indication to the visual data delivery system (e.g., to the server) (¶ [0047]: “At step 406, there is a request to display an image on client. For example, client 312 may be running image processing software. A user may interact with the software to perform an image processing task on a medical image. The software may initiate an image processing request that is relayed to the operating system of the client 312. This image processing request may include information about the image to be processed, such as image quality. The image processing request may also include information about the nature of the image processing request, such as to zoom or pan the image, for example. The image processing request may be communicated from the client 312 to the server 302.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”),
wherein the 2D image plane adjustment indication indicates the required rotation and/or translation of the 2D image plane (¶ [0047]: “At step 406, there is a request to display an image on client. For example, client 312 may be running image processing software. A user may interact with the software to perform an image processing task on a medical image. The software may initiate an image processing request that is relayed to the operating system of the client 312. This image processing request may include information about the image to be processed, such as image quality. The image processing request may also include information about the nature of the image processing request, such as to zoom or pan the image, for example. The image processing request may be communicated from the client 312 to the server 302.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”);
receive, via the interface circuitry (As shown in FIG. 2, all communications between server 102 and client 104 is via communication pathway 106. Likewise, in FIG. 3, all communications between server 302 and client 312 is via communication pathway 305.), second 2D image data representing a second sequence of 2D images of the body from the visual data delivery system (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”),
wherein the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane (¶ [0012]: “A variety of techniques are known for processing 3D image data into 2D image data. These techniques include multi-planar reformatting (MPR), maximum (or minimum) intensity projection (MIP), and volume rendering (VR). In MPR processing, a 3D volume may be processed to obtain a 2D slice that may be different than the slices obtained by a medical imaging system. For example, an application incorporating MPR functionality may allow a user to rotate a displayed image at any angle and centered at any location within the volume. Thus, MPR allows a clinician to view the anatomy from any of a variety of positions and angles.” ¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.”); and
display, using the display screen, the second sequence of 2D images (¶ [0082]: “Next, at step 808, the image data, initially stored on the server, is processed by the server, and transmitted to the client. The processed image data is a 2D image of the fetus that appears 3D. The 2D image is displayed on the client at step 810. Next, at step 812, the user views the image of the fetus. The user, however, wishes to view a different angle of the fetus. Accordingly, the user opens a control panel in the software application, and enters a new viewing angle. The software interprets the interaction, and generates an image processing request. The image with the new view angle will require that the image data be reprocessed with MPR techniques. The software application recognizes that this type of interaction may require substantial processing resources. Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7. At step 808, the image data is then processed in accordance with the image processing request. The images are then displayed on the client in step 810.” ).
Whereas MUSSACK is not entirely explicit as to, VERONESI discloses a display system (e.g., FIG. 1), the display system comprising a processing unit, a display screen (¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.” ) and interface circuitry (¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.”), wherein the processing unit is configured to:
receive, via the interface circuitry, first 2-dimensional (2D) image data representing a first sequence of 2D images of a body (e.g., ¶ [0020]: “intersection information” ¶ [0031]: “generating one or more 2D slice image(s) from the 3D ultrasound dataset”) from a visual data delivery system (e.g., FIG. 1: Processor 116 and/or Memory 120) (¶ [0020]: “FIG. 2 shows a high-level flow chart illustrating an example method 200 for displaying intersection information on an ultrasound image according to an embodiment. Method 200 will be described herein with reference to the system and components depicted in FIG. 1, though it should be understood that the method may be applied to other systems and components without departing from the scope of the present disclosure. Method 200 may be carried out by processor 116, and may be stored as executable instructions in non-transitory memory of the processor 116.” ¶ [0021]: “Method 200 begins at 205. At 205, method 200 includes acquiring ultrasound data for a region of interest (ROI). In some examples, the ROI may comprise a structure or object, for example an organ such as a human heart or a region thereof. The ultrasound data may comprise a 3D ultrasound dataset or in some examples a 4D ultrasound dataset. For example, the ultrasound data may comprise a volume of data including 3D color Doppler data over time, such as over one or more heart cycles (e.g., ultrasound echocardiography data), and may be stored in a memory device.” ¶ [0024]: “At 215, method 200 includes receiving an indication of the ROI. The indication of the ROI may comprise an indication received, from a user via a user interface such as user interface 115, of a particular region or organ of interest. In some examples, the indication may include two or more ROIs. For example, the user may desire to image the heart and more specifically the mitral valve of the heart, and therefore may indicate that the ROI includes both the heart and the mitral valve.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0030]: “As another example, one or more planes may be automatically identified and indicated based on the indication of the ROI. For example, standard views may be predetermined for a given ROI, and the indication of the slice positions corresponding to the standard views may be automatically retrieved and/or generated based on the indication of the ROI as well as the 3D mesh model. In other words, the slice positions or planes may be located at fixed pre-determined positions relative to the data volume or the ultrasound probe. For example, two orthogonal slice planes corresponding to the azimuth and elevation planes of the acquired ultrasound ROI may be positioned such that the planes intersect the center of the data volume. As another example, three slice planes may be rotated about a common axis (such as the probe axis) where the planes are by default oriented to provide visualization of a four chamber view, a two chamber view, and a long axis view of the left ventricle of the heart. The user may or may not modify the position and orientation of these planes.” ¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.” ¶ [0043]: “The first 2D slice 330 and the second 2D slice 350 may comprise different 2D views of the 3D ultrasound dataset.” NOTE: The 4D dataset clearly comprises a sequence of 3D image data, and, as such, the 2D slice images of the 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).),
wherein the 2D images (e.g., ¶ [0020]: “intersection information” ¶ [0031]: “generating one or more 2D slice image(s) from the 3D ultrasound dataset”) are generated from a sequence of 3-dimensional (3D) images (¶ [0021]: “The ultrasound data may comprise a 3D ultrasound dataset or in some examples a 4D ultrasound dataset. For example, the ultrasound data may comprise a volume of data including 3D color Doppler data over time, such as over one or more heart cycles (e.g., ultrasound echocardiography data), and may be stored in a memory device.” ¶ [0039]: “It should be appreciated that although the method 200 is described with regard to 3D ultrasound datasets, method 200 may be applied to four-dimensional datasets comprising three spatial dimensions plus time.” ¶ [0017]: “A memory 120 is included for storing processed volumes of acquired data. In an exemplary embodiment, the memory 120 is of sufficient capacity to store at least several seconds worth of volumes of ultrasound data. The volumes of data are stored in a manner to facilitate retrieval thereof according to its order or time of acquisition.” ¶ [0021]: “a region of interest (ROI).” ¶ [0024]: “a particular region or organ of interest.” NOTE: A 4D data set comprising 3D image data over time is a sequence of 3D image data.) of the body in a 2D image plane (¶ [0002]: “to visualize intersections between volume data and planes.” ¶ [0002]: “display one or more 2D slice planes reconstructed from a 3D ultrasound data volume,” ¶ [0011]: “planes within the 3D dataset for rendering.” ¶ [0020]: “FIG. 2 shows a high-level flow chart illustrating an example method 200 for displaying intersection information on an ultrasound image according to an embodiment. Method 200 will be described herein with reference to the system and components depicted in FIG. 1, though it should be understood that the method may be applied to other systems and components without departing from the scope of the present disclosure. Method 200 may be carried out by processor 116, and may be stored as executable instructions in non-transitory memory of the processor 116.” ¶ [0029]: “a determination is made as to the coordinates of the plane(s) through the 3D volume dataset” ¶ [0021]: “Method 200 begins at 205. At 205, method 200 includes acquiring ultrasound data for a region of interest (ROI). In some examples, the ROI may comprise a structure or object, for example an organ such as a human heart or a region thereof. The ultrasound data may comprise a 3D ultrasound dataset or in some examples a 4D ultrasound dataset. For example, the ultrasound data may comprise a volume of data including 3D color Doppler data over time, such as over one or more heart cycles (e.g., ultrasound echocardiography data), and may be stored in a memory device.” ¶ [0024]: “At 215, method 200 includes receiving an indication of the ROI. The indication of the ROI may comprise an indication received, from a user via a user interface such as user interface 115, of a particular region or organ of interest. In some examples, the indication may include two or more ROIs. For example, the user may desire to image the heart and more specifically the mitral valve of the heart, and therefore may indicate that the ROI includes both the heart and the mitral valve.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0030]: “As another example, one or more planes may be automatically identified and indicated based on the indication of the ROI. For example, standard views may be predetermined for a given ROI, and the indication of the slice positions corresponding to the standard views may be automatically retrieved and/or generated based on the indication of the ROI as well as the 3D mesh model. In other words, the slice positions or planes may be located at fixed pre-determined positions relative to the data volume or the ultrasound probe. For example, two orthogonal slice planes corresponding to the azimuth and elevation planes of the acquired ultrasound ROI may be positioned such that the planes intersect the center of the data volume. As another example, three slice planes may be rotated about a common axis (such as the probe axis) where the planes are by default oriented to provide visualization of a four chamber view, a two chamber view, and a long axis view of the left ventricle of the heart. The user may or may not modify the position and orientation of these planes.” ¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.” ¶ [0043]: “The first 2D slice 330 and the second 2D slice 350 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 330 and 350 with the 3D image volume 305 may be visually depicted in the 3D mesh model 320. For example, the plane corresponding to the first 2D slice 330 may be represented as a line or intersection 335 on the 3D mesh model 320, while the plane corresponding to the second 2D slice 350 may be represented as a line or intersection 355 on the 3D mesh model 320. The intersections 335 and 355 may also be displayed or overlaid on the 3D image volume 305.” ¶ [0051]: “The first 2D slice 430, the second 2D slice 450, and the third 2D slice 470 may comprise different 2D views of the 3D ultrasound dataset. The intersections between the 2D slices 430, 450, and 470 with the 3D image volume 405 may be visually depicted in the 3D mesh model 420.” NOTE: The 4D dataset clearly comprises a sequence of 3D image data, and, as such, the 2D slice images of the 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).),
display, using the display screen (e.g., FIG. 1: “Display device” 118; ¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.”), the first sequence of 2D images (¶ [0031]: “At 235, method 200 includes generating one or more 2D slice image(s) from the 3D ultrasound dataset based on the indication of the one or more slice positions. The one or more 2D slice images may be displayed on a display device, such as display device 118, alongside the 3D mesh model and the 3D image volume.”);
receive, via the interface circuitry (e.g., ¶ [0015]: “The processor 116 is also in electronic communication with a display device 118, and the processor 116 may process the data into images for display on the display device 118.”), one or more 3D models to the display system (¶ [0023]: “At 210, method 200 includes rendering a 3D image volume from the ultrasound data. For example, a 3D image volume or 3D image may be reconstructed from the 3D ultrasound dataset using any suitable 3D volumetric image reconstruction technique. In some embodiments, the 3D volume dataset is displayed in real-time, for example, on the display device 118.” ¶ [0027]: “At 225, method 200 includes generating a 3D mesh model of the ROI. The 3D mesh model of the ROI may be retrieved from a digital repository of 3D mesh models corresponding to different organs or anatomies, as an illustrative and non-limiting example. As another example, the method may generate a 3D mesh model of the segmented ROI obtained at 220. In yet another example, the method may retrieve a generic 3D mesh model of the ROI from a digital repository and fit the mesh model to the segmented ROI obtained at 220.” ¶ [0028]: “Method 200 further includes coupling the 3D mesh model to the 3D image. For example, one or more points of the 3D mesh model may be linked to one or more corresponding voxels of the ROI in the 3D image. In this way, the scale and orientation of the 3D mesh model may correspond to the scale and orientation of the ROI in the 3D image. The coupling may be carried out automatically or in some examples may be carried out with assistance from user input. For example, the method may automatically identify corresponding points in the 3D mesh model and the 3D image, or a user may manually identify, via the user interface 115 for example, the corresponding points in the 3D mesh model and the 3D image.”), and
display, using the display screen, the one or more 3D models for the body or a part of the body shown in the first sequence of 2D images (¶ [0023]: “At 210, method 200 includes rendering a 3D image volume from the ultrasound data. For example, a 3D image volume or 3D image may be reconstructed from the 3D ultrasound dataset using any suitable 3D volumetric image reconstruction technique. In some embodiments, the 3D volume dataset is displayed in real-time, for example, on the display device 118.” ¶ [0042]: “As depicted, the 3D mesh model 320 of the ROI 308 may be displayed adjacent to the 3D image volume 305 in the graphical user interface 300.”); or
receive, via the interface circuitry, an indication of an identified type of body or type of body part, or an indication of one or more predetermined 3D models corresponding to the identified type of body or type of body part, to the display system, and display, using the display screen, one or more predetermined 3D models corresponding to of the identified type of body or type of body part, or an indication of one or more predetermined 3D models corresponding to the identified type of body or type of body part (NOTE: The alternative to this limitation has been met and, as such, this limitation is not required to be given patentable weight.);
receive, via the interface circuitry (¶ [0014]: “A user interface 115 may be used to control operation of the ultrasound imaging system 100, including controlling the input of patient data, changing a scanning or display parameter, and the like. The user interface 115 may include a graphical user interface configured for display on a display device 118. The graphical user interface may include information to be output to a user (such as ultrasound images, patient data, etc.) and may also include menus or other elements through which a user may enter input to the computing system. In examples described in more detail below with respect to FIGS. 2-4, the user interface may receive inputs from a user indicating, for example, adjustments to the position of planes to be imaged. The user interface 115 may include one or more of the following: a rotary, a mouse, a keyboard, a trackball, a touch-sensitive display, hard keys linked to specific actions, soft keys that may be configured to control different functions, and a graphical user interface.”), a user input from a user of the display system (¶ [0014]: “with respect to FIGS. 2-4, the user interface may receive inputs from a user indicating, for example, adjustments to the position of planes to be imaged.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0036]: “At 255, method 200 includes determining if one or more of the 2D slice(s) will be adjusted. Determining if one or more of the 2D slices will be adjusted comprises determining if user input regarding a new position for a 2D slice on the 3D mesh model and/or the 3D image volume. For example, the user may select an intersection displayed on the 3D mesh model and manually move or adjust the position of the intersection within the 3D mesh model, for example via the user interface 115. In some examples, the user may adjust the display of the 3D mesh model, for example by scaling the display of the 3D mesh model and/or rotating or otherwise changing the displayed perspective of the 3D mesh model. Changing the display of the 3D mesh model makes it easier for the user to navigate the 3D volume to select a new position for a 2D slice.” ¶ [0037] If there is no user input relating to a new position for a 2D slice, then there are no slice adjustments (“NO”), and method 200 returns. However, if there is user input relating to a new position for a 2D slice, there is a slice adjustment (“YES”), and method 200 continues to 260. At 260, method 200 includes receiving an indication of a new slice position, for example via the user interface 115.” ¶ [0038]: “At 265, method 200 includes generating and displaying a new 2D slice from the ultrasound dataset at the indicated position. At 270, method 200 includes displaying the updated intersections between the new 2D slice, the 3D mesh model, and the 3D image. Method 200 then returns.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.”),
wherein the user input indicates a required rotation and/or translation of the 2D image plane (¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image through the 3D model and/or 3D image dataset. Thus, this limitation is inherently taught by VERONESI.);
send, via the interface circuitry (e.g., ¶ [0014]: “A user interface 115” ), a 2D image plane adjustment indication to the visual data delivery system (¶ [0014]: “A user interface 115 may be used to control operation of the ultrasound imaging system 100, including controlling the input of patient data, changing a scanning or display parameter, and the like. The user interface 115 may include a graphical user interface configured for display on a display device 118. The graphical user interface may include information to be output to a user (such as ultrasound images, patient data, etc.) and may also include menus or other elements through which a user may enter input to the computing system. In examples described in more detail below with respect to FIGS. 2-4, the user interface may receive inputs from a user indicating, for example, adjustments to the position of planes to be imaged. The user interface 115 may include one or more of the following: a rotary, a mouse, a keyboard, a trackball, a touch-sensitive display, hard keys linked to specific actions, soft keys that may be configured to control different functions, and a graphical user interface.” ¶ [0029]: “Continuing at 230, method 200 includes receiving an indication of one or more slice positions. For example, one or more planes or slices in the 3D mesh model may be selected or indicated by a user, for example via the user interface 115. For example, the user may manually move or position virtual slices on the screen to selected different views to display. Thus, based on one or more user-selected or—marked planes, which may be selected image views, a determination is made as to the coordinates of the plane(s) through the 3D volume dataset corresponding to the location in the 3D mesh model. In some examples, the voxels within the 3D volume dataset corresponding to the user-selected plane(s) are determined.” ¶ [0036]: “At 255, method 200 includes determining if one or more of the 2D slice(s) will be adjusted. Determining if one or more of the 2D slices will be adjusted comprises determining if user input regarding a new position for a 2D slice on the 3D mesh model and/or the 3D image volume. For example, the user may select an intersection displayed on the 3D mesh model and manually move or adjust the position of the intersection within the 3D mesh model, for example via the user interface 115. In some examples, the user may adjust the display of the 3D mesh model, for example by scaling the display of the 3D mesh model and/or rotating or otherwise changing the displayed perspective of the 3D mesh model. Changing the display of the 3D mesh model makes it easier for the user to navigate the 3D volume to select a new position for a 2D slice.” ¶ [0037] If there is no user input relating to a new position for a 2D slice, then there are no slice adjustments (“NO”), and method 200 returns. However, if there is user input relating to a new position for a 2D slice, there is a slice adjustment (“YES”), and method 200 continues to 260. At 260, method 200 includes receiving an indication of a new slice position, for example via the user interface 115.” ¶ [0038]: “At 265, method 200 includes generating and displaying a new 2D slice from the ultrasound dataset at the indicated position. At 270, method 200 includes displaying the updated intersections between the new 2D slice, the 3D mesh model, and the 3D image. Method 200 then returns.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.”),
wherein the 2D image plane adjustment indication indicates the required rotation and/or translation of the 2D image plane (¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image through the 3D model and/or 3D image dataset. Thus, this limitation is inherently taught by VERONESI.);
receive, via the interface circuitry, second 2D image data representing a second sequence of 2D images of the body from the visual data delivery system (¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” ¶ [0065] In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: As already noted herein-above, a 4D dataset clearly comprises a sequence of 3D image data, and, as such, the 2D slice images of a 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).),
wherein the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane (¶ [0060]: “receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0063]: “receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image relative to the 3D model and/or corresponding 3D dataset. Thus, this limitation is inherently taught by VERONESI.); and
display, using the display screen (e.g., ¶ [0062]: “system comprises a display device and a processor communicatively coupled to the display device,”), the second sequence of 2D images (¶ [0038] At 265, method 200 includes generating and displaying a new 2D slice from the ultrasound dataset at the indicated position. At 270, method 200 includes displaying the updated intersections between the new 2D slice, the 3D mesh model, and the 3D image. Method 200 then returns.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” ¶ [0065] In a first example of the system, the system further comprises a user interface communicatively coupled to the processor, wherein the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to adjust display on the display device of the 3D model responsive to user input received via the user interface. In a second example of the system optionally including the first example, the processor is further configured with instructions in the non-transitory memory that when executed cause the processor to receive, via the user interface, a user adjustment of a position of the intersection between the 2D image and the 3D model, generate a second 2D image based on the user adjustment, and output the second 2D image to the display device.” NOTE: As already noted herein-above, the 2D slice images of a 4D dataset (i.e., a 3D dataset over time) clearly comprise a sequence of 2D slice images of the ROI (i.e., the body).).
Thus, in order to obtain a more versatile a distributed medical image processing system having the cumulative features and/or functionalities taught by MUSSACK and VERONESI, it would have been obvious to one of ordinary skill in the art to have modified the distributed medical image processing system taught by MUSSACK such that the display system processor (i.e., the client in MUSSACK) is also configured to: receive first 2D image data representing a first sequence of 2D images of a body from the visual data delivery system (i.e., the server in MUSSACK), wherein the 2D images are generated from a sequence of 3D images of the body in a 2D plane; receive one or more 3D models to the display system and display the one or more 3D models for the body or a part of the body shown in the first sequence of images; and receive second image data representing a second sequence of images of the body, wherein the second sequence of 2D images are images of the body in the rotated and/or translated 2D image plane, and display the second sequence of images, as taught by VERONESI.
Regarding claim 11 (depends on claim 10), whereas MUSSACK may not be entirely explicit as to, VERONESI further teaches:
wherein the user input indicating the required rotation and/or translation of the 2D image plane is received as a user input that rotates and/or translates a displayed 3D model (¶ [0036]: “At 255, method 200 includes determining if one or more of the 2D slice(s) will be adjusted. Determining if one or more of the 2D slices will be adjusted comprises determining if user input regarding a new position for a 2D slice on the 3D mesh model and/or the 3D image volume. For example, the user may select an intersection displayed on the 3D mesh model and manually move or adjust the position of the intersection within the 3D mesh model, for example via the user interface 115. In some examples, the user may adjust the display of the 3D mesh model, for example by scaling the display of the 3D mesh model and/or rotating or otherwise changing the displayed perspective of the 3D mesh model. Changing the display of the 3D mesh model makes it easier for the user to navigate the 3D volume to select a new position for a 2D slice.” ¶ [0047]: “As discussed hereinabove with regard to FIG. 2, a user may rotate or scale the 3D mesh model 320 to obtain a different perspective or view of the ROI. The user may also select one or more of the intersections 335 and 355, and adjust the position of the intersections 335 and 355. Accordingly, the corresponding 2D image slice will be updated with a new 2D image slice rendered from ultrasound data at the adjusted intersection positions. Further, the intersection information may be updated on each image.” ¶ [0060]: “In another embodiment, a method comprises acquiring a three-dimensional (3D) ultrasound dataset of a region of interest (ROI), generating a 3D model of the ROI, generating a 3D image and a two-dimensional (2D) image slice from the 3D ultrasound dataset, displaying the 3D image, the 2D image slice, and the 3D model, displaying an intersection between the 3D image and the 2D image slice on the 3D model, receiving user input for an adjusted slice position with respect to the 3D model, generating a second 2D image slice from the 3D ultrasound dataset based on the adjusted slice position, and replacing the display of the 2D image slice with the second 2D image slice.” NOTE: By necessity, a user adjustment of a position of the intersection between the slice plane of the 2D image and the 3D model requires a rotation, a translation and/or a rotation and translation of the slice plane of the 2D image relative to the 3D model and/or corresponding 3D dataset. Thus, this limitation is inherently taught by VERONESI.).
Regarding claim 1, claim 1 is directed to the computer-implemented method of operating the visual data delivery system of claim 7 and, as such, is rejected for the same reasons applied above in the rejection of as claim 7.
Regarding claim 2 (depends on claim 1), claim 2 is directed to the computer-implemented method of operating the visual data delivery system of claim 8 and, as such, is rejected for the same reasons applied above in the rejection of as claim 8.
Regarding claim 3 (depends on claim 1), claim 3 is directed to the computer-implemented method of operating the visual data delivery system of claim 9 and, as such, is rejected for the same reasons applied above in the rejection of as claim 9.
Regarding claim 4, claim 4 is directed to the computer-implemented method of operating the display system of claim 10 and, as such, is rejected for the same reasons applied above in the rejection of as claim 10.
Regarding claim 5 (depends on claim 4), claim 5 is directed to the computer-implemented method of operating the display system of claim 11 and, as such, is rejected for the same reasons applied above in the rejection of as claim 11.
Regarding claim 6 (depends on claim 1), claim 6 is directed to computer program product comprising a computer readable medium having computer readable code for executing the method of claim 1 and, a such is rejected for the same reasons applied above to claim 1.
Regarding claim 12 (depends on claim 11), MUSSACK discloses:
wherein the first 2D image data is sent to the display system (e.g., Client 104 in FIGS. 1-2 and/or Client 312 in FIG. 3) via a data connection (e.g., communication pathway 106 in FIGS. 1-2 and/or 305 in FIG. 3) between the visual data delivery system (e.g., Server 102 in FIGS. 1-2 and/or Server 302 in FIG. 3) and the display system (¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: Image “data” is transmitted over the communication pathway, and, as such, the communication pathway is therefore a “data connection.”),
wherein the data connection has a transfer bandwidth less than a bandwidth required to stream the 3D images (¶ [0005]: “One of the factors that may influence the design of a network is bandwidth. In distributed networks with relatively low bandwidth, it may be desirable to choose a design that reduces network traffic. For example, in a low bandwidth network, it may be desirable to provide a server-centric network.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0040]: “Bandwidth of communication pathway 305 may factor in determining how system resources are allocated in order to display medical images on client 312. Bandwidth may be estimated or measured by either of client 312 or server 302. For example, client monitor 314 or server monitor 304 may estimate bandwidth by any of a number of known techniques. One technique for estimating bandwidth is to communicate a test packet of known size across communication pathway 305, and measure a time for completion of test packet communication.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.” ¶ [0057]: “As an illustrative example, embodiments of the present application may be used in the following manner. On a distributed network 300, a client 312 displays a medical image on display 214. Image data may be stored locally, at client 312, or at server 302. The client 312 provides an application that allows a user to edit or alter the displayed image. A user interacts with client 312 through user interface 216. The user can choose from a variety of image processing tasks to edit or alter the displayed image with the image processing application including: annotation; contrast adjustment; brightness adjustment; grayscale adjustment; pan; zoom; rotation; 3D processing. In this illustrative example, user chooses "contrast adjustment," by directing user interface 216 accordingly. Meanwhile, both client monitor 314 and server monitor 310 have been tracking system resources and bandwidth. Server monitor 310 has been tracking server CPU 202 load, and the most recent indication is that server CPU is relatively unloaded--e.g. the server CPU 202 is over 95% idle. Client monitor 314 determined at startup that client 312 has a relatively slow processor and clock speed. Client monitor 314 has also been tracking bandwidth of communication pathway 305, and the most recent indication is that bandwidth is relatively low. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. Recommendation provider 306 determines that an efficient image processing mode is to only allocate client resources 316 to adjust the contrast of the image. The recommendation provider 306 also determines that the client 312 can perform this image processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides the recommendations that client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 automatically accepts the recommendations, and proceeds to process the image by adjusting the contrast per the user's directive.” ¶ [0058]: “Continuing the illustrative example, the user next seeks to perform a relatively complicated image processing task. The user seeks to view a different angle of the image. This type of image processing involves the relatively intensive MPR technique. Additionally, the user seeks to retain the contrast adjustment previously applied when viewing a new angle of the image. Both client monitor 314 and server monitor 310 observe no significant changes with system resources and bandwidth. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. In this case, recommendation provider 306 determines that an efficient image processing mode is to allocate a portion of server resources 308 and a portion client resources 316 to process the image. Server resources 308 can most efficiently perform MPR processing, while client resources 316 can most efficiently perform subsequent contrast adjustment processing. The recommendation provider 306 also determines that system resources can perform processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides recommendations that certain server resources 308 and certain client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 and server 302 automatically accept the recommendations, and proceed to process the image per the user's directive.” ¶ [0082]: “Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7.” NOTE: As noted in ¶ [0015], “if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network.” Thus, in the instances where the network bandwidth is relatively low (e.g., ¶ [0049]: and ¶ [0057]-[0058]), the connection pathway clearly has a transfer bandwidth less than a bandwidth required to stream the 3D images (i.e., real-time transfer of 3D images).).
Regarding claim 13 (depends on claim 12), MUSSACK discloses:
wherein the data connection comprises a network connection (¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.”).
Regarding claim 14 (depends on claim 4), MUSSACK discloses:
wherein the first 2D image data is received by the display system via a data connection (e.g., ¶ [0030]: “communication pathway 106”) between the visual data delivery system (e.g., ¶ [0030]: “server 102,”) and the display system (e.g., ¶ [0030]: “client 104,”) (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716.” ¶ [0030]: “Distributed network 100 may include a server 102, a client 104, and a communication pathway 106. The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: Image “data” is transmitted over the communication pathway, and, as such, the communication pathway is therefore a “data connection.”),
wherein the data connection has a transfer bandwidth less than a bandwidth required to stream the 3D images (¶ [0005]: “One of the factors that may influence the design of a network is bandwidth. In distributed networks with relatively low bandwidth, it may be desirable to choose a design that reduces network traffic. For example, in a low bandwidth network, it may be desirable to provide a server-centric network.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0040]: “Bandwidth of communication pathway 305 may factor in determining how system resources are allocated in order to display medical images on client 312. Bandwidth may be estimated or measured by either of client 312 or server 302. For example, client monitor 314 or server monitor 304 may estimate bandwidth by any of a number of known techniques. One technique for estimating bandwidth is to communicate a test packet of known size across communication pathway 305, and measure a time for completion of test packet communication.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.” ¶ [0057]: “As an illustrative example, embodiments of the present application may be used in the following manner. On a distributed network 300, a client 312 displays a medical image on display 214. Image data may be stored locally, at client 312, or at server 302. The client 312 provides an application that allows a user to edit or alter the displayed image. A user interacts with client 312 through user interface 216. The user can choose from a variety of image processing tasks to edit or alter the displayed image with the image processing application including: annotation; contrast adjustment; brightness adjustment; grayscale adjustment; pan; zoom; rotation; 3D processing. In this illustrative example, user chooses "contrast adjustment," by directing user interface 216 accordingly. Meanwhile, both client monitor 314 and server monitor 310 have been tracking system resources and bandwidth. Server monitor 310 has been tracking server CPU 202 load, and the most recent indication is that server CPU is relatively unloaded--e.g. the server CPU 202 is over 95% idle. Client monitor 314 determined at startup that client 312 has a relatively slow processor and clock speed. Client monitor 314 has also been tracking bandwidth of communication pathway 305, and the most recent indication is that bandwidth is relatively low. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. Recommendation provider 306 determines that an efficient image processing mode is to only allocate client resources 316 to adjust the contrast of the image. The recommendation provider 306 also determines that the client 312 can perform this image processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides the recommendations that client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 automatically accepts the recommendations, and proceeds to process the image by adjusting the contrast per the user's directive.” ¶ [0058]: “Continuing the illustrative example, the user next seeks to perform a relatively complicated image processing task. The user seeks to view a different angle of the image. This type of image processing involves the relatively intensive MPR technique. Additionally, the user seeks to retain the contrast adjustment previously applied when viewing a new angle of the image. Both client monitor 314 and server monitor 310 observe no significant changes with system resources and bandwidth. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. In this case, recommendation provider 306 determines that an efficient image processing mode is to allocate a portion of server resources 308 and a portion client resources 316 to process the image. Server resources 308 can most efficiently perform MPR processing, while client resources 316 can most efficiently perform subsequent contrast adjustment processing. The recommendation provider 306 also determines that system resources can perform processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides recommendations that certain server resources 308 and certain client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 and server 302 automatically accept the recommendations, and proceed to process the image per the user's directive.” ¶ [0082]: “Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7.” NOTE: As noted in ¶ [0015], “if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network.” Thus, in the instances where the network bandwidth is relatively low (e.g., ¶ [0049]: and ¶ [0057]-[0058]), the connection pathway clearly has a transfer bandwidth less than a bandwidth required to stream the 3D images (i.e., real-time transfer of 3D images).).
Regarding claim 15 (depends on claim 14), MUSSACK discloses:
wherein the data connection comprises a network connection (¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.”).
Regarding claim 16 (depends on claim 7), MUSSACK discloses:
wherein the interface circuitry (e.g., ¶ [0030]: “communication pathway 106”) enables a data connection between the visual data delivery system (e.g., ¶ [0030]: “server 102,”) and the display system (e.g., ¶ [0030]: “client 104,”) (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716.” ¶ [0030]: “Distributed network 100 may include a server 102, a client 104, and a communication pathway 106. The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: Image “data” is transmitted over the communication pathway, and, as such, the communication pathway is therefore a “data connection.”),
wherein the data connection has a transfer bandwidth less than a bandwidth required to stream the 3D images (¶ [0005]: “One of the factors that may influence the design of a network is bandwidth. In distributed networks with relatively low bandwidth, it may be desirable to choose a design that reduces network traffic. For example, in a low bandwidth network, it may be desirable to provide a server-centric network.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0040]: “Bandwidth of communication pathway 305 may factor in determining how system resources are allocated in order to display medical images on client 312. Bandwidth may be estimated or measured by either of client 312 or server 302. For example, client monitor 314 or server monitor 304 may estimate bandwidth by any of a number of known techniques. One technique for estimating bandwidth is to communicate a test packet of known size across communication pathway 305, and measure a time for completion of test packet communication.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.” ¶ [0057]: “As an illustrative example, embodiments of the present application may be used in the following manner. On a distributed network 300, a client 312 displays a medical image on display 214. Image data may be stored locally, at client 312, or at server 302. The client 312 provides an application that allows a user to edit or alter the displayed image. A user interacts with client 312 through user interface 216. The user can choose from a variety of image processing tasks to edit or alter the displayed image with the image processing application including: annotation; contrast adjustment; brightness adjustment; grayscale adjustment; pan; zoom; rotation; 3D processing. In this illustrative example, user chooses "contrast adjustment," by directing user interface 216 accordingly. Meanwhile, both client monitor 314 and server monitor 310 have been tracking system resources and bandwidth. Server monitor 310 has been tracking server CPU 202 load, and the most recent indication is that server CPU is relatively unloaded--e.g. the server CPU 202 is over 95% idle. Client monitor 314 determined at startup that client 312 has a relatively slow processor and clock speed. Client monitor 314 has also been tracking bandwidth of communication pathway 305, and the most recent indication is that bandwidth is relatively low. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. Recommendation provider 306 determines that an efficient image processing mode is to only allocate client resources 316 to adjust the contrast of the image. The recommendation provider 306 also determines that the client 312 can perform this image processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides the recommendations that client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 automatically accepts the recommendations, and proceeds to process the image by adjusting the contrast per the user's directive.” ¶ [0058]: “Continuing the illustrative example, the user next seeks to perform a relatively complicated image processing task. The user seeks to view a different angle of the image. This type of image processing involves the relatively intensive MPR technique. Additionally, the user seeks to retain the contrast adjustment previously applied when viewing a new angle of the image. Both client monitor 314 and server monitor 310 observe no significant changes with system resources and bandwidth. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. In this case, recommendation provider 306 determines that an efficient image processing mode is to allocate a portion of server resources 308 and a portion client resources 316 to process the image. Server resources 308 can most efficiently perform MPR processing, while client resources 316 can most efficiently perform subsequent contrast adjustment processing. The recommendation provider 306 also determines that system resources can perform processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides recommendations that certain server resources 308 and certain client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 and server 302 automatically accept the recommendations, and proceed to process the image per the user's directive.” ¶ [0082]: “Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7.” NOTE: As noted in ¶ [0015], “if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network.” Thus, in the instances where the network bandwidth is relatively low (e.g., ¶ [0049]: and ¶ [0057]-[0058]), the connection pathway clearly has a transfer bandwidth less than a bandwidth required to stream the 3D images (i.e., real-time transfer of 3D images).).
Regarding claim 17 (depends on claim 16), MUSSACK discloses:
wherein the data connection comprises a network connection (¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.”).
Regarding claim 18 (depends on claim 16), MUSSACK discloses:
wherein the data connection is enabled according to a wired or wireless protocol ¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.”).
Regarding claim 19 (depends on claim 10), MUSSACK discloses:
wherein the interface circuitry (e.g., ¶ [0030]: “communication pathway 106”) enables a data connection between the visual data delivery system (e.g., ¶ [0030]: “server 102,”) and the display system (e.g., ¶ [0030]: “client 104,”) (¶ [0042]: “3D image data may be processed by a server to form 2D image data. Processing may include techniques such as MPR, MIP, VR, and/or other image processing techniques that convert 3D data into 2D data. The 2D image data may be further processed to form image data for display at step 706. For example, 2D image data may be further processed by adjusting grayscale, contrast, and/or brightness. At step 708, 2D image data may be sent from a server, and then received by a client at step 710.” ¶ [0043]: “Next, the 2D image data may be sent from a server at step 714 and received at a client in step 716.” ¶ [0030]: “Distributed network 100 may include a server 102, a client 104, and a communication pathway 106. The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.” NOTE: Image “data” is transmitted over the communication pathway, and, as such, the communication pathway is therefore a “data connection.”),
wherein the data connection has a transfer bandwidth less than a bandwidth required to stream the 3D images (¶ [0005]: “One of the factors that may influence the design of a network is bandwidth. In distributed networks with relatively low bandwidth, it may be desirable to choose a design that reduces network traffic. For example, in a low bandwidth network, it may be desirable to provide a server-centric network.” ¶ [0015]: “Techniques for converting 3D data into a 2D displayable image, such as MPR, MIP, and VR, may be useful to clinicians. In addition, such techniques may also significantly reduce the size of data. A 2D displayable image may be a fraction of the size of a 3D data volume. However, techniques such as MPR, MIP, and VR may consume a substantial amount of processing resources. If processing resources are not readily available, imaging performance may become slow or degraded. Similarly, if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network. Moreover, other factors such as image quality of both 3D data and processed 2D data may impact the performance of an image display system.” ¶ [0040]: “Bandwidth of communication pathway 305 may factor in determining how system resources are allocated in order to display medical images on client 312. Bandwidth may be estimated or measured by either of client 312 or server 302. For example, client monitor 314 or server monitor 304 may estimate bandwidth by any of a number of known techniques. One technique for estimating bandwidth is to communicate a test packet of known size across communication pathway 305, and measure a time for completion of test packet communication.” ¶ [0049]: “In example 2, the server 302 is relatively unloaded, meaning that the server CPU is not overly burdened by other pending tasks. Furthermore, the client 312 processing speed is relatively slow, and the bandwidth of the communications pathway is relatively low. In this scenario some simple processing tasks, like pan & zoom may be performed by client 312, while advanced tasks may be reserved for server 302. In this scenario all processing tasks are recommended to be performed primarily on the server 302. The recommendation in example 2 may correspond to, anticipate, or trigger a process flow similar to that depicted in the second branch 750 of FIG. 7.” ¶ [0057]: “As an illustrative example, embodiments of the present application may be used in the following manner. On a distributed network 300, a client 312 displays a medical image on display 214. Image data may be stored locally, at client 312, or at server 302. The client 312 provides an application that allows a user to edit or alter the displayed image. A user interacts with client 312 through user interface 216. The user can choose from a variety of image processing tasks to edit or alter the displayed image with the image processing application including: annotation; contrast adjustment; brightness adjustment; grayscale adjustment; pan; zoom; rotation; 3D processing. In this illustrative example, user chooses "contrast adjustment," by directing user interface 216 accordingly. Meanwhile, both client monitor 314 and server monitor 310 have been tracking system resources and bandwidth. Server monitor 310 has been tracking server CPU 202 load, and the most recent indication is that server CPU is relatively unloaded--e.g. the server CPU 202 is over 95% idle. Client monitor 314 determined at startup that client 312 has a relatively slow processor and clock speed. Client monitor 314 has also been tracking bandwidth of communication pathway 305, and the most recent indication is that bandwidth is relatively low. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. Recommendation provider 306 determines that an efficient image processing mode is to only allocate client resources 316 to adjust the contrast of the image. The recommendation provider 306 also determines that the client 312 can perform this image processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides the recommendations that client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 automatically accepts the recommendations, and proceeds to process the image by adjusting the contrast per the user's directive.” ¶ [0058]: “Continuing the illustrative example, the user next seeks to perform a relatively complicated image processing task. The user seeks to view a different angle of the image. This type of image processing involves the relatively intensive MPR technique. Additionally, the user seeks to retain the contrast adjustment previously applied when viewing a new angle of the image. Both client monitor 314 and server monitor 310 observe no significant changes with system resources and bandwidth. The server monitor 310 provides server CPU loading information as monitor data to recommendation provider 306. The client monitor 314 provides client processing speed and bandwidth information as monitor data to recommendation provider 306. In this case, recommendation provider 306 determines that an efficient image processing mode is to allocate a portion of server resources 308 and a portion client resources 316 to process the image. Server resources 308 can most efficiently perform MPR processing, while client resources 316 can most efficiently perform subsequent contrast adjustment processing. The recommendation provider 306 also determines that system resources can perform processing task with sufficient speed only on a low quality image. The recommendation provider 306 provides recommendations that certain server resources 308 and certain client resources 316 should be allocated for this task, and that the image should be of low quality. Client 312 and server 302 automatically accept the recommendations, and proceed to process the image per the user's directive.” ¶ [0082]: “Consequently, the process flow is routed back to step 806. At step 806, it is determined that the network conditions have worsened, and the bandwidth is now relatively low. Therefore, in accordance with example 2 of FIG. 5, most of the 3D image processing is allocated to be performed on the server, while simpler tasks are allocated to be performed by the client. This type of allocation is shown, for example, in branch 750 of FIG. 7.” NOTE: As noted in ¶ [0015], “if a network has relatively low bandwidth, it may take a relatively long time to transfer 3D image data across the network.” Thus, in the instances where the network bandwidth is relatively low (e.g., ¶ [0049]: and ¶ [0057]-[0058]), the connection pathway clearly has a transfer bandwidth less than a bandwidth required to stream the 3D images (i.e., real-time transfer of 3D images).).
Regarding claim 20 (depends on claim 10), MUSSACK discloses:
wherein the data connection comprises a network connection (¶ [0030]: “The communication pathway 106 has an associated bandwidth. The bandwidth of communication pathway 106 may be uniform across the entire communication pathway 106, or it may vary along various segments. For example, a communication pathway 106 may include a combination of various types of networks having various bandwidths, such as a copper wire twisted-pair network and an optical network. A communication pathway 106 may include a local area network (LAN), wide area network (WAN), wired network, wireless network, optical network, and the like, or any combination thereof. Similarly, a communication pathway 106 may include various network elements, such as routers, repeaters, switches, hubs, splitters, couplers, intermediary computers, or the like.”).
Response to Arguments
Applicant's arguments filed February 24, 2026 with respect to the amended claims have been considered but are moot in view of the new ground(s) of rejection necessitated by said amendments.
Conclusion
At present, it is not apparent to the examiner which part of the application could serve as a basis for new and allowable claims. However, should the applicant nevertheless regard some particular matter as patentable, the examiner encourages applicant to appropriately amend the claims to include such matter and to indicate in the REMARKS the difference(s) between the prior art and the claimed invention as well as the significance thereof.
Furthermore, should applicant decide to amend the claims, examiner respectfully requests that the applicant please indicate in the REMARKS from which page(s), line(s) or claim(s) of the originally filed application that any amendments are derived. See MPEP § 2163(II)(A) (There is a strong presumption that an adequate written description of the claimed invention is present in the specification as filed, Wertheim, 541 F.2d at 262, 191 USPQ at 96; however, with respect to newly added or amended claims, applicant should show support in the original disclosure for the new or amended claims.).
Action is Final
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Relevant Prior Art
The following prior art, although not relied upon, is made of record since it is considered pertinent to applicant's disclosure:
Hu et al. (US 6621918) discloses a teleradiology system that provides the capability of rendering and studying of a remotely located volume data without requiring transmission of the entire data to the user's local computer. The system comprises: receiving station under the control of a user; transmitting station; the connecting network; a user interface with functionality of controlling volume data rendering, transmission, and display; and the interface with patient data source.
Saito et al. (US 6683933) discloses a three-dimensional image processing system enables multiple users at distant locations employing ordinary personal computers to construct and observe three-dimensional images simultaneously. In a network environment, a three-dimensional image processing device acquires image data and incorporates it into three-dimensional voxels based on instructions from the computers. The device operates on the three-dimensional voxels using object space domain, opacity and color parameters to construct three-dimensional data and operates on the three-dimensional data using projection processing parameters to construct three-dimensional images. The system sets projection processing parameters for constructing three-dimensional images from three-dimensional data and displays three-dimensional images which may be routed to the various computers via the network.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT PEREN who can be reached by telephone at (571) 270-7781, or via email at vincent.peren@uspto.gov. The examiner can normally be reached on Monday-Friday from 10:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KING POON, can be reached at telephone number (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/VINCENT PEREN/
Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights