DETAILED ACTION
This Office action is responsive to communications filed on 08/15/2025. Claims 1, 3-5, 8-11 have been amended. Presently, Claims 1-11 remain pending and are hereinafter examined on the merits.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
A substantial amount of the previous 35 U.S.C. 112(b) are withdrawn in view of the amendments filed on 08/15/2025. However, Claim 4: the phrase “the structure type” in lines 4-5 remains indefinite. It is unclear if the phrase refers to a structure type of the plurality of structure types in line 2 or if the phrase refers to the structure type of the structure define in line 12 of claim 1. The phrase is interpreted as the structure types of the plurality of structure types.
The previous claim objections are withdrawn in view of the amendments filed on 08/15/2025.
Applicant's arguments filed 08/15/2025 have been fully considered but they are not persuasive. Under the broadest reasonable interpretation the claim term “space compound likelihood image” is not expressly defined in the claims are shown to a have special definition in the specification. The ‘space compound” reasonably encompasses an image produce by spatially compounding information derived from multiple ultrasonic views/frames (e.g., different steer angles or scan positions). Likewise the claims do not further define the likelihood distribution other than only stating “a likelihood distribution”. That is the claims do not differentiate the likelihood distribution from combined cited references Patton in view of Karube in view of Takada.
Applicant’s first contends that Patton’s “blended image 580” is merely an image that depicts both an anatomical structure and an interventional instrument created from images 530 and 560, and that it is therefore not a “space compound likelihood image”. This argument is unpersuasive because the rection does not rely on Patton along to teach every limitation with respect to the generating of the space compound likelihood image. Patton is cited form multi-angle acquisition, and the overall image formation pipeline that spatially combines outputs to detect the same target within anatomy from multiple views. Karube supplies the missing plurality of per-image likelihood maps and their integration across multiple scans/views, and Takada supplies the “likelihood distribution of existence” of the same target within each frame. Modifying Patton’s multiple single frame instrument indication with Karube in view of Takada, and then using Pattons compounding improvement yields the synthesis that claim 1 requires. The Applicants assertions of the data objects that Patton uses (e.g., echo data, images 530/560) does not negate the obviousness of modifying Patton’s pipeline to accept and combine per frame likelihood images as taught by Karube in view of Takada. The claim does not preclude merging likelihood derived outputs with underlying image content; it only requires that the synthesis generate a “space compound likelihood image”.
The Applicant further contends that Karube’s per-scan probability maps relate to different finding. This is a mischaracterization of Karube’s teachings. Karube teaches generating probability maps for each scan and integrating the multiple probability maps generated correspond to multiple scans of the same wound, ¶Abstract. Under the broadest reasonably interpretation “same target” is the class of object/structure the system is locating. For any one selected target/finding of interest, Karube disclosure entails a plurality of per-scan probability maps for that same target/finding which are then integrated; the fact that Karaube also discloses doing this synthesis for multiple different finding does not detract from the teach that for one chosen target, multiple per-scan likelihood maps are computed and combined. Thus, Karube teaches the required “same target”.
The Applicant further contends that Takada does not teach or reasonably suggest the concept of combining plural likelihood images each indicating a likelihood distribution with respect to a same target and generated from respective different ultrasound images. This argument is unpersuasive. The rejection does not rely on Takada to each the cross image synthesis, that is relied on Patton’s compounding framework, and Karube integration of per-scan probability maps. Takada is cited for the “likelihood distribution existence of the same target” within each image, (i.e., the likelihood values overs pixels/regions for the target, performed frame-by-frame). Takada merely images the same region of interest that has likelihood values above a threshold (i.e., detection regions) within the region of interest, combines these likelihod values, and the selects the region with the highest likelihood as the target location.
To an extent, the Applicant argues the “likelihood image” must be a stand alone probability map or that the space compound likelihood image excludes any merger with grayscale/tissue information, which the claims don’t preclude this interpretation. The claim merely only requires that synthesis of the plurality of likelihood images to generate a space compound likelihood image. In sum, the Applicant’s arguments rely on stricter constructions that what the claims actually recite. For these reasons, the 35 USC § 103 is maintained.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as failing to set forth the subject matter which the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the applicant regards as the invention.
Claim 4: the phrase “the structure type” in lines 4-5 render the claim indefinite. It is unclear if the phrase refers to a structure type of the plurality of structure types in line 2 or if the phrase refers to the structure type of the structure define in line 12 of claim 1. The phrase is interpreted as the structure types of the plurality of structure types. Appropriate correction is required.
The dependent claims of the above rejected claims are rejected due to their dependency.
Claim Objections
The following claims are objected to because of the following informalities and should recite:
Claim 4: “the same target”. Appropriate correction is required.
Claim 4: “the same target”. Appropriate correction is required.
Claim 5: “the same target”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Patton et al (US 2020/0178927 A1) in view of Karube (WO 2021/256019 A1) in view of Takada et al (US 2022/0313220 A1).
Claim 1: Patton discloses, An ultrasonic diagnostic apparatus comprising: ([Abstract], ‘an ultrasound imaging system is configured to receive a set of ultrasound images of a target anatomical region.’)
transmitter-receiver circuity (a transmit/receive switch 46, FIG. 2) that outputs a reception signal acquired from an ultrasonic probe; ([0019], ‘the amplified transmit signals may be supplied to the transducer probe 12 through the transmit/receive switch 46, which disconnects or shields sensitive receive electronics from the transmit signals at the time they are delivered to the transducer probe 12. After the signals are transmitted, the transmit/receive switch 46 may connect the receive electronics to the transducer elements to detect the corresponding electronic echo signals created when the returning acoustic waves impinge upon the transducer elements.’)
a signal processor (image processor 58, FIG. 2) that processes the reception signal acquired from the ultrasonic probe; and (FIG. 2, [0022], ‘Images produced by the image processor 58 from the received signals may be displayed on a display 60.’; [0024], ‘The anatomy image created by the image processor 58 may be stored in memory to be combined with echo data for one or more of the needle frames that are created to locate an interventional instrument.’)
an ultrasound image acquisition process that acquires, via the transmit-receive circuitry and the signal processor, a plurality ultrasonic images generated by ultrasonic scanning using ultrasonic beams having respective different steer angles; (FIG. 4; [0019], ‘the amplified transmit signals may be supplied to the transducer probe 12 through the transmit/receive switch 46, which disconnects or shields sensitive receive electronics from the transmit signals at the time they are delivered to the transducer probe 12. After the signals are transmitted, the transmit/receive switch 46 may connect the receive electronics to the transducer elements to detect the corresponding electronic echo signals created when the returning acoustic waves impinge upon the transducer elements.’; FIG. 2, [0022], ‘Images produced by the image processor 58 from the received signals may be displayed on a display 60.’; [0024], ‘The anatomy image created by the image processor 58 may be stored in memory to be combined with echo data for one or more of the needle frames that are created to locate an interventional instrument.’; ¶0036, ‘In the example of FIG. 4, the ultrasound imaging system may capture three frames 410, 411, and 412 to be used to determine the tissue frame. In particular embodiments, the three frames 410, 411, and 412 are taken at different transmit beam directions, but directed at the same anatomical structure.’)
a hardware processor (processor 40) that, under control of a stored program, controls processes comprising: (¶0067, ‘A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.’)
a target identification process ([0017], ‘the ultrasound system may include a processor 40 having a built-in or external memory (not shown) containing instructions that are executable by the processor to operate the ultrasound imaging system’; [0024], ‘the processor 40 may be programmed to create a composite image of the tissue being examined and an interventional instrument being introduced into the tissue.’; [0052], ‘the classifier algorithm may be further trained to identify which of the shallow, medium, and steep needle frames may be most appropriate for creation of the composite image.’) that performs segmentation processing on each of the plurality of ultrasonic images based on a structure type of a structure therein, to thereby generate a likelihood image (singular) of a same target within the ultrasonic image among the plurality of ultrasonic images; and ([0026], ‘the echo data for each of the needle frames created from the transmissions at the different transmit angles may be analyzed for the presence of an interventional instrument […] Various instrument detection algorithms may be used. As an example and not by way of limitation, the images may be analyzed for the presence of a linear segment of pixels that are much brighter (e.g. greater amplitude) than adjacent pixels thereby indicating the presence of a strong linear reflector. The length of the segments that may represent an interventional instrument may vary and in some embodiments, may be curved if the interventional instrument itself is curved or bends when the instrument is inserted […], each segment of bright pixels may be scored to indicate how likely the segment represents an interventional instrument. As an example and not by way of limitation, such a score may be adjusted by the length of the bright pixels above a certain threshold, how straight or linear the segment of pixels is, how much contrast is present between the bright pixels and the adjacent pixels, how strong the edges around the segment of bright pixels are as determined by a gradient or other edge-detection operations, etc’; [0031], ‘the processor may be programmed to identify segments of pixel data from one or more of the needle frames created from the transmissions taken at the various transmit angles that have a score that indicates the pixels likely represent an interventional instrument.’; [0038], ‘[Upon receiving an indication from the user that an interventional instrument is present, and the direction from which the interventional instrument is being introduced, the ultrasound imaging system may capture multiple needle frames 540, 541, and 542, from different angles. In the example of FIG. 5, needle frame 540 is captured at a steer angle of +24 degrees; needle frame 541 is captured at a steer angle of +32 degrees; and needle frame 542 is captured at a steer angle of +40 degrees. In particular embodiments, the angles for the shallow, medium, and steep needle frames may be predetermined by the ultrasound imaging system, or preset by the user.’; [0052], ‘a classifier algorithm may be trained only to detect whether an interventional instrument is present. In particular embodiments, a classifier algorithm may also be trained to identify left or right entry of the interventional instrument. As an example and not by way of limitation, a classifier algorithm may analyze an ultrasound image and return a score ranging from 0 to 1, where 0 indicates “left” and 1 indicates “right.” In this example, scores in between 0 and 1 may represent a likelihood that the interventional instrument is to the left or right. In particular embodiments, a classifier algorithm may have a first step of determining whether an interventional instrument is present, then in a second step, if an interventional instrument is present, then determining a score corresponding to whether there is a left-side or right-side entry of the interventional instrument, as described above.’)
generate a space compound likelihood image. (560 - ¶0039 ‘FIG. 5, the ultrasound imaging system then determine which of needle frames 540, 541, and 542 are appropriate for determining the linear structure. In particular embodiments, the ultrasound imaging system determines which of the needle frames most prominently shows the presence of an interventional instrument by choosing the frame with the highest linear structure score [...] The ultrasound imaging system may then generate an enhanced linear structure image 560 that depicts the interventional instrument (at the selected needle frame angle) with a masked region defined around the linear structure within the same field of view as composite tissue frame 530. At step 570, the ultrasound imaging system may merge the composite tissue frame 530 with the enhanced linear structure 560 to generate a blended image 580, which depicts the interventional instrument and its relative position to the imaged anatomical structures.’; ¶0052, ‘[0052], ‘a classifier algorithm may be trained only to detect whether an interventional instrument is present. In particular embodiments, a classifier algorithm may also be trained to identify left or right entry of the interventional instrument. As an example and not by way of limitation, a classifier algorithm may analyze an ultrasound image and return a score ranging from 0 to 1, where 0 indicates “left” and 1 indicates “right.” In this example, scores in between 0 and 1 may represent a likelihood that the interventional instrument is to the left or right. In particular embodiments, a classifier algorithm may have a first step of determining whether an interventional instrument is present, then in a second step, if an interventional instrument is present, then determining a score corresponding to whether there is a left-side or right-side entry of the interventional instrument, as described above.’)
Patton fails to disclose: to thereby generate a plurality of likelihood images each indicating an existence of a same target within its corresponding ultrasonic image from among the plurality of the ultrasonic images; and
a likelihood image synthesizing process that synthesizes the plurality of likelihood images generated from the plurality of the ultrasonic images to generate a space compound likleihood image
However, Karube in the context of calculating a presence probability of a finding related to a wound portion from each of the ultrasound images discloses: to thereby generate a plurality of likelihood images each indicating an existence of a same target within its corresponding ultrasonic image from among the plurality of the ultrasonic images; (¶0009, ‘when multiple scans are performed on the same wound, the probability map generation unit can generate a probability map for each scan, integrate the multiple probability maps generated corresponding to the multiple scans, and display the integrated probability map on the monitor.’; ¶0012, ‘calculates the presence probability of a finding related to the wound from each of the plurality of frames of ultrasound images, and generates a three-dimensional probability map of the findings based on the acquired position information of the ultrasound probe and the calculated presence probability.’; ¶0027, ‘the probability calculation unit 9 calculates the probability of the presence of findings related to the wound by performing image recognition on each of the multiple frames of ultrasound images generated by the image generation unit 4.’; ¶0029, ‘Furthermore, the probability calculation unit 9 can calculate the probability of the presence of multiple findings for each pixel of the ultrasound image in each frame, for example, by using a deep learning method such as the so-called U-net. In this case, for example, for one pixel, the probability that the pixel corresponds to an unclear layer structure A1, the probability that the pixel corresponds to a Cobblestone-like pattern A2, the probability that the pixel corresponds to a Cloud-like pattern A3, and the probability that the pixel corresponds to a pattern A4 in which liquid accumulation is observed are each calculated.’)
-Karube generation of likelihood (probability) maps by computing pixel-wise probability’s across multiples scans of finds, corresponds to a plurality of likelihood images of a target within it corresponding ultrasonic image from among the plurality of ultrasonic images.
a likelihood image synthesizing process that synthesizes the plurality of likelihood images generated from the plurality of the ultrasonic images to generate a space compound likelihood image (¶0035, ‘When integrating the probability maps of the findings, the probability map generating unit 10 assigns one finding to each pixel based on the value of the presence probability of each finding plotted at each pixel… the probability map generating unit 10 can assign to the pixel the finding corresponding to the greatest existence probability among these four existence probabilities.’; ¶0053, ‘In the following step S5, the probability map generator 10 integrates the three-dimensional probability maps of the respective findings generated in step S4 to generate a three-dimensional integrated probability map.’)
-Karube discloses a likelihood image synthesizing process that synthesis the plurality of likelihood images generated from the plurality of ultrasonic images to generate the three-dimensional integrated probability map via per-pixel finding selection and spatial integrations across scans. ¶0035-¶0053.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method and system of Patton to include the teachings of Karube. The motivation to do this yields predictable results such as enabling users to accurately detect the type of findings, ¶0014 of Karube.
Patton in view of Karube fail to disclose when generating a plurality of likelihood images each indicating a likelihood distribution of existence of a same target within its corresponding ultrasonic image from amount the plurality of the ultrasonic images.
However, Takada in the context of likelihood targets in ultrasonic images discloses, to thereby generate a plurality of likelihood images each indicating a likelihood distribution of existence of a same target within its corresponding ultrasonic image from amount the plurality of the ultrasonic images. (¶0128, ‘the processing circuitry 180 performs image segmentation related to the lacteal gland region on the ultrasonic image 1330, and generates the segmentation image 1320. The segmentation image 1320 shows five segmented regions 1321 to 1325, with which “skin”, “fat”, “lacteal gland”, “tumor”, and “pectoral major muscle” are associated, respectively.’; ¶0076, ‘The first ultrasonic image data acquired may be video data including two or more frames.’ ¶0058, ‘estimates the position of an examination target included in ultrasonic image data by applying a trained model to the ultrasonic image data, and outputs an estimation result. As an example of estimating the position of an examination target, when a region in the ultrasonic image data exists in which the likelihood is equal to or greater than a threshold, it is estimated that an examination target is included in that region. In this case, the result of estimation includes, for example, at least one region estimated to include an examination target (this region may be referred to as a “detection region” or a “unit of detection”). When the estimation likelihood of all the regions in the ultrasonic image data does not exceed (or fall below) a threshold, it is estimated that the image data does not include an examination target.’;; ¶0095, ‘FIG. 8 is a diagram illustrating an example of the likelihood of the units of detection included in the multiple detection areas…¶0096, ‘For the detection area 721, the processing circuitry 180 calculates a total value “2.07” combining the likelihood “0.71” of the unit A1 of detection, the likelihood “0.73” of the unit A2 of detection, and the likelihood “0.63” of the unit A3 of detection. For the detection area 722, the processing circuitry 180 calculates a total value “2.77” combining the likelihood “0.91” of the unit B1 of detection, the likelihood “0.93” of the unit B2 of detection, and the likelihood “0.93” of the unit B3 of detection. Likewise, for the detection area 723, the processing circuitry 180 calculates a total value “1.24” combining the likelihood “0.61” of the unit C1 of detection and the likelihood “0.63” of the unit C2 of detection. After calculating a total likelihood value of each detection area, the processing circuitry 180 selects the detection area 722 having the highest likelihood total value’)
- Takada teaches applying a trained model to “ultrasound image date” which is expressly disclosed as comprising “including two or more frames” ¶0076. The model is applied to the image data indicating a likelihood distribution of existence of a target within the ultrasonic image, further discussing examples with respect to one ultrasound image. The first ultrasonic image data acquired may be video data including two or more frames. Each frame in the video data is considered an image. Since the image data includes two or more frames, (i.e., two or more images), the likelihood distribution indicated for the image data is considered a likelihood distribution of existence of a target for each of the two or more frames.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the target identification process of Patton such that the each of the generated plurality of ultrasonic images has an indication of a likelihood distribution of existence of a same target within its corresponding ultrasonic image from among the plurality of ultrasonic images as taught by Takada. The motivation to do this yields predictable results such as improve the estimation of the target structure within the ultrasound data using trained models, enabling precise ROI localization based on likelihood scores.
Claim 2: Modified Patton discloses all the elements above in claim 1, Patton discloses, wherein
the space compound likelihood image (560) is applied as an enhancement map (Enhanced Compounding 570, FIG. 5; [0039], ‘At step 570, the ultrasound imaging system may merge the composite tissue frame 530 with the enhanced linear structure 560 to generate a blended image 580, which depicts the interventional instrument and its relative position to the imaged anatomical structures.’) of a region including the target in a space compound ultrasonic image (Blended Image 580, FIG. 5, -the space compound likelihood image, 560, is applied as an enhancement map via the enhancing compound 570 including the target in a space compound ultrasonic image, 580.) generated by synthesizing the plurality of the ultrasonic images ([0002], ‘the user or the ultrasound system may take a number of images at different angles to vary the beam direction, so that the best image of the needle can be determined and used.’; [0008], ‘FIG. 5 depicts an example diagram of an ultrasound imaging system that may automatically select an optimal needle frame from a plurality of steer angles, and produce and display a blended image of tissue and an interventional instrument.’; [0010], ‘FIG. 7 depicts an example diagram of an ultrasound imaging system that may automatically detect the presence of an interventional instrument as well as the direction of entry, automatically select an optimal needle frame from a plurality of steer angles, and produce and display a blended image of tissue and an interventional instrument.’; [0038], ‘FIG. 5 depicts another example system for creating a blended image combining a composite tissue frame and a linear structure corresponding to an interventional instrument. In the example of FIG. 5, the ultrasound imaging system may capture a plurality of needle frames, and determine which is most suitable for combining into the blended image. In FIG. 5, similarly to FIG. 4, the ultrasound imaging system may capture three frames 510, 511, and 512, at multiple angles suitable for combining into a B-mode image. At step 520, the ultrasound imaging system may combine the frames 510, 511, and 512 into a composite tissue frame 530 for spatial compounding. Upon receiving an indication from the user that an interventional instrument is present, and the direction from which the interventional instrument is being introduced, the ultrasound imaging system may capture multiple needle frames 540, 541, and 542, from different angles.’; [0039], ‘FIG. 5, the ultrasound imaging system then determine which of needle frames 540, 541, and 542 are appropriate for determining the linear structure. In particular embodiments, the ultrasound imaging system determines which of the needle frames most prominently shows the presence of an interventional instrument by choosing the frame with the highest linear structure score [...] The ultrasound imaging system may then generate an enhanced linear structure image 560 that depicts the interventional instrument (at the selected needle frame angle) with a masked region defined around the linear structure within the same field of view as composite tissue frame 530. At step 570, the ultrasound imaging system may merge the composite tissue frame 530 with the enhanced linear structure 560 to generate a blended image 580, which depicts the interventional instrument and its relative position to the imaged anatomical structures.’)
Claim 3: Modified Patton discloses all the elements above in claim 1, Patton fails to disclose: wherein the likelihood image synthesizing process synthesizes the plurality of the likelihood images by using an image synthesis method set for the structure type of the structure to generate the space compound likelihood image.
However, Karube is relied upon above discloses: wherein the likelihood image synthesizing process synthesizes the plurality of the likelihood images by using an image synthesis method set for the structure type of the structure to generate the space compound likelihood image. (¶0035, ‘When integrating the probability maps of the findings, the probability map generating unit 10 assigns one finding to each pixel based on the value of the presence probability of each finding plotted at each pixel… the probability map generating unit 10 can assign to the pixel the finding corresponding to the greatest existence probability among these four existence probabilities.’; ¶0053, ‘In the following step S5, the probability map generator 10 integrates the three-dimensional probability maps of the respective findings generated in step S4 to generate a three-dimensional integrated probability map.’)
-Karube discloses a likelihood image synthesizing process that synthesis the plurality of likelihood images generated from the plurality of ultrasonic images to generate the three-dimensional integrated probability map via an image synthesis method (i.e., per-pixel finding selection and spatial integrations across scans). ¶0035-¶0053.
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the likelihood image synthesizing process of modified Patton to use an image synthesis method set for the structure type of the structure to generate the space compound likelihood image from the plurality of likelihood images of Karube. The motivation to do this yields predictable results such as enabling users to accurately detect the type of findings, ¶0014 of Karube.
Claim 4: Modified Patton discloses all the elements above in claim 3, Patton fails to disclose: wherein when a plurality of structure types are set, the likelihood image synthesizing process synthesizes the plurality of the likelihood images by using the image synthesis method set for the structure type of the target existing in a pixel region having been segmented in the segmentation processing to generate the space compound likelihood image.
However, Karube is relied upon above discloses: wherein when a plurality of structure types are set, (¶0050, ‘Step S3…The probability calculation unit 9 can calculate the probability of the presence of multiple findings for each pixel of multiple frames of ultrasound images, for example, by using a deep learning method such as so-called U-net. As a result, each pixel of ultrasound images of multiple frames has a probability of the presence of multiple findings, such as the probability that the pixel corresponds to an unclear layer structure A1, the probability that the pixel corresponds to a Cobblestone-like pattern A2, the probability that the pixel corresponds to a Cloud-like pattern A3, or the probability that the pixel corresponds to a pattern A4 in which liquid accumulation is observed, as shown in Figures 4 to 7.’) the likelihood image synthesizing process synthesizes the plurality of the likelihood images by using the image synthesis method set for the structure type of the target existing in a pixel region having been segmented in the segmentation processing (¶0029, ‘the probability calculation unit 9 can calculate the probability of the presence of multiple findings for each pixel of the ultrasound image in each frame, for example, by using a deep learning method such as the so-called U-net. In this case, for example, for one pixel, the probability that the pixel corresponds to an unclear layer structure A1, the probability that the pixel corresponds to a Cobblestone-like pattern A2, the probability that the pixel corresponds to a Cloud-like pattern A3, and the probability that the pixel corresponds to a pattern A4 in which liquid accumulation is observed are each calculated.’) to generate the space compound likelihood image. (¶0052, ‘At this time, the probability map generator 10 plots the existence probability obtained in step S3 for each of the multiple findings for each pixel arranged three-dimensionally based on the position information of the ultrasound probe 21 stored together with the multiple frames of ultrasound images, thereby visualizing the existence probability of each finding, thereby generating a three-dimensional probability map of each finding.’; ¶0053, ‘In the following step S5, the probability map generator 10 integrates the three-dimensional probability maps of the respective findings generated in step S4 to generate a three-dimensional integrated probability map.’)
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the likelihood image synthesizing process of modified Patton such that wherein when a plurality of structure types are set, the likelihood image synthesizing process synthesizes the plurality of the likelihood images by using the image synthesis method set for the structure type of the target existing in a pixel region having been segmented in the segmentation process to generate the space compound likelihood image as taught by Karube. The motivation to do this yields predictable results such as enabling users to accurately detect the type of findings, ¶0014 of Karube.
Claim 7: Modified Patton discloses all the elements above in claim 1, Patton fails to disclose:, wherein the likelihood image synthesizing process synthesizes the plurality of likelihood images by using an image synthesis method set by a user to generate the space compound likelihood image. (¶0035, ‘When integrating the probability maps of the findings, the probability map generating unit 10 assigns one finding to each pixel based on the value of the presence probability of each finding plotted at each pixel… the probability map generating unit 10 can assign to the pixel the finding corresponding to the greatest existence probability among these four existence probabilities.’; ¶0053, ‘In the following step S5, the probability map generator 10 integrates the three-dimensional probability maps of the respective findings generated in step S4 to generate a three-dimensional integrated probability map.’; ¶0061, ‘Also, for example, the user can switch between displaying the integrated probability map and the individual probability maps of each finding via the input device 17, allowing them to be displayed separately. This allows the user to more easily and accurately grasp the types of multiple findings and their distribution.’; ¶0065, ‘Furthermore, in cases where the user checks the integrated probability map or the individual probability maps of each finding in step S6 and determines that an additional location other than the location scanned by the ultrasound probe 21 in step S2 needs to be scanned, the processing of steps S1 to S6 can be performed again in response to a user instruction via the input device 17.’)
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the likelihood image synthesizing process of modified Patton to include using an imaging synthesis method set by a user to generate the space compound likelihood image of Karube for the advantage of allowing the user to more easily and accurately grasp the types of multiple finding and their distribution, ¶0061 of Karube.
Claim 8: Modified Patton discloses all the elements above in claim 1, Patton discloses, wherein
the target identification process performs the segmentation processing ([0026], ‘the echo data for each of the needle frames created from the transmissions at the different transmit angles may be analyzed for the presence of an interventional instrument […] Various instrument detection algorithms may be used. As an example and not by way of limitation, the images may b e analyzed for the presence of a linear segment of pixels that are much brighter (e.g. greater amplitude) than adjacent pixels thereby indicating the presence of a strong linear reflector. The length of the segments that may represent an interventional instrument may vary and in some embodiments, may be curved if the interventional instrument itself is curved or bends when the instrument is inserted […], each segment of bright pixels may be scored to indicate how likely the segment represents an interventional instrument. As an example and not by way of limitation, such a score may be adjusted by the length of the bright pixels above a certain threshold, how straight or linear the segment of pixels is, how much contrast is present between the bright pixels and the adjacent pixels, how strong the edges around the segment of bright pixels are as determined by a gradient or other edge-detection operations, etc’; [0031], ‘the processor may be programmed to identify segments of pixel data from one or more of the needle frames created from the transmissions taken at the various transmit angles that have a score that indicates the pixels likely represent an interventional instrument.’; [0052], ‘a classifier algorithm may be trained only to detect whether an interventional instrument is present. In particular embodiments, a classifier algorithm may also be trained to identify left or right entry of the interventional instrument. As an example and not by way of limitation, a classifier algorithm may analyze an ultrasound image and return a score ranging from 0 to 1, where 0 indicates “left” and 1 indicates “right.” In this example, scores in between 0 and 1 may represent a likelihood that the interventional instrument is to the left or right. In particular embodiments, a classifier algorithm may have a first step of determining whether an interventional instrument is present, then in a second step, if an interventional instrument is present, then determining a score corresponding to whether there is a left-side or right-side entry of the interventional instrument, as described above.’) based on the structure type of the structure in each of the plurality of ultrasonic images (FIG. 4, 5, & 7) by using an identification model learned by machine learning. ([0016], ‘As an example and not by way of limitation, a trained neural network may analyze one or more ultrasound images to determine whether an interventional instrument is present, and the appropriate angle to capture the interventional instrument so that a needle frame may be combined with the anatomy image to create a composite frame. ’; [0050], ‘the ultrasound imaging system may utilize one or more classifier algorithms to determine whether an interventional instrument is present in the anatomy image, without utilizing a needle frame image. In particular embodiments, a classifier algorithm may rely on artificial intelligence, machine-learning, neural networks, or any other suitable algorithm to analyze the ultrasound images. As an example and not by way of limitation, a classifier algorithm may detect tissue warping in the anatomy image that occurs over time due to the movement of the interventional instrument affecting the surrounding tissue.’; [0051], ‘the trained classifier algorithms may be based on a neural network. As may be well understood in the art, a neural network includes a plurality of individual “neuron” algorithms, wherein each neuron is trained to detect specific inputs and output a value when the inputs are detected. A neural network may have a large number of neurons working in parallel, as well as many layers of neurons that iteratively receive inputs from the previous layer and provide an output to the following layer. As an example and not by way of limitation, a neural network may be trained to detect left or right entry of an interventional instrument by inputting a large number of images, as well as information indicating for each image whether an interventional instrument is present, which direction, and which needle frame angle is appropriate. By training the neurons so that the outputs based on the input images correspond to the information known about each image, the neural network may be trained to receive images in the future and perform the classifier algorithms discussed herein. In particular embodiments, using a neural network to develop the classifier algorithms may allow training process to be fully automated, given a sufficient number of labeled training images. In particular embodiments, given enough training images and feedback for training, a neural network may detect any other inputs in the ultrasound images that may be relevant for detecting an interventional instrument and its direction of entry, without requiring an explicit input from a user to look for such an input. In particular embodiments, using a trained classifier algorithm may provide a technical improvement to the ultrasound imaging system in automatically detecting the presence of an interventional instrument, its direction, and selecting a needle frame angle for imaging the interventional instrument.’)
Claim 9: Modified Patton discloses all the elements above in claim 8, Patton discloses, wherein
the identification model is a neural network. ([0016], ‘As an example and not by way of limitation, a trained neural network may analyze one or more ultrasound images to determine whether an interventional instrument is present, and the appropriate angle to capture the interventional instrument so that a needle frame may be combined with the anatomy image to create a composite frame. ’; [0050], ‘the ultrasound imaging system may utilize one or more classifier algorithms to determine whether an interventional instrument is present in the anatomy image, without utilizing a needle frame image. In particular embodiments, a classifier algorithm may rely on artificial intelligence, machine-learning, neural networks, or any other suitable algorithm to analyze the ultrasound images. As an example and not by way of limitation, a classifier algorithm may detect tissue warping in the anatomy image that occurs over time due to the movement of the interventional instrument affecting the surrounding tissue.’; [0051], ‘the trained classifier algorithms may be based on a neural network. As may be well understood in the art, a neural network includes a plurality of individual “neuron” algorithms, wherein each neuron is trained to detect specific inputs and output a value when the inputs are detected. A neural network may have a large number of neurons working in parallel, as well as many layers of neurons that iteratively receive inputs from the previous layer and provide an output to the following layer. As an example and not by way of limitation, a neural network may be trained to detect left or right entry of an interventional instrument by inputting a large number of images, as well as information indicating for each image whether an interventional instrument is present, which direction, and which needle frame angle is appropriate. By training the neurons so that the outputs based on the input images correspond to the information known about each image, the neural network may be trained to receive images in the future and perform the classifier algorithms discussed herein. In particular embodiments, using a neural network to develop the classifier algorithms may allow training process to be fully automated, given a sufficient number of labeled training images. In particular embodiments, given enough training images and feedback for training, a neural network may detect any other inputs in the ultrasound images that may be relevant for detecting an interventional instrument and its direction of entry, without requiring an explicit input from a user to look for such an input. In particular embodiments, using a trained classifier algorithm may provide a technical improvement to the ultrasound imaging system in automatically detecting the presence of an interventional instrument, its direction, and selecting a needle frame angle for imaging the interventional instrument.’)
Claim 10: Patton discloses, A method for controlling an ultrasonic diagnostic apparatus, the method comprising: ([Abstract], ‘an ultrasound imaging system is configured to receive a set of ultrasound images of a target anatomical region. The set of ultrasound images is combined to create a composite tissue frame. The ultrasound imaging system determines whether an interventional instrument is present within the ultrasound images based on a set of trained classification algorithms based on the set of ultrasound images and the composite tissue frame.’)
outputting a reception signal acquired from an ultrasonic probe; ([0019], ‘the amplified transmit signals may be supplied to the transducer probe 12 through the transmit/receive switch 46, which disconnects or shields sensitive receive electronics from the transmit signals at the time they are delivered to the transducer probe 12. After the signals are transmitted, the transmit/receive switch 46 may connect the receive electronics to the transducer elements to detect the corresponding electronic echo signals created when the returning acoustic waves impinge upon the transducer elements.’)
processing the reception signal acquired from the ultrasonic probe; (FIG. 2, [0022], ‘Images produced by the image processor 58 from the received signals may be displayed on a display 60.’; [0024], ‘The anatomy image created by the image processor 58 may be stored in memory to be combined with echo data for one or more of the needle frames that are created to locate an interventional instrument.’)
acquiring a plurality of ultrasonic images generated by ultrasonic scanning using ultrasonic beams having respective different steer angles; (FIG. 4; [0019], ‘the amplified transmit signals may be supplied to the transducer probe 12 through the transmit/receive switch 46, which disconnects or shields sensitive receive electronics from the transmit signals at the time they are delivered to the transducer probe 12. After the signals are transmitted, the transmit/receive switch 46 may connect the receive electronics to the transducer elements to detect the corresponding electronic echo signals created when the returning acoustic waves impinge upon the transducer elements.’; FIG. 2, [0022], ‘Images produced by the image processor 58 from the received signals may be displayed on a display 60.’; [0024], ‘The anatomy image created by the image processor 58 may be stored in memory to be combined with echo data for one or more of the needle frames that are created to locate an interventional instrument.’; ¶0036, ‘In the example of FIG. 4, the ultrasound imaging system may capture three frames 410, 411, and 412 to be used to determine the tissue frame. In particular embodiments, the three frames 410, 411, and 412 are taken at different transmit beam directions, but directed at the same anatomical structure.’)
performing segmentation processing on each of the plurality of ult