DETAILED ACTION
This office action is in response to the communication received on 09/29/2026 concerning application no. 18/703,422 filed on 04/22/2024.
Claims 1 and 13-16 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/29/2026 has been entered.
Claims 1 and 13-16 are pending.
Response to Arguments
Applicant's arguments filed 09/29/2026 have been fully considered but they are not persuasive.
Regarding the 112(b) rejections of the claims’ recitation of fixed images and the associated elements, Applicant argues “The Examiner asserted ambiguity regarding how a single fixed image relates to a plurality of fixed images individually displayed at different fixed positions based on user input applied multiple times. As set forth in paragraphs 70-74 and 101, when an operator manipulates the ultrasound probe to a site, the operator actuates the input device (such as a foot pedal or gesture control), causing the system to display a "fixed image" where that composite ultrasound image slice is pinned with transparency at that spatial location in mixed reality. As the operator continues the scan to adjacent tissue areas and applies the input multiple times, each respective composite image slice is captured and pinned at its respective location. Thus, the system maintains a plurality of fixed images, each individually displayed at its respective fixed position. The claim language explicitly recites this physical and temporal operational sequence.”
Examiner disagrees. Applicant is reminded that the claims are rejected under 112(b) and not 112(a). The claims do not recite the language of the paragraphs 70-74 and 101. The claim is assessed based on the language of the claim itself. That language is indefinite. The claim fails to clearly establish the relationship of the fixed images to the preceding fixed image that is established to be continuously maintained for the composite image. Additionally, it is well-known in the art that an image is a “a visual representation of something”1. The language is unclear as to what manner the fixed images are able to individually display a plurality of fixed positions. Furthermore, the preceding claim element establishes a suer input is the basis of capture of the composite image. The instant element establishes the input occurring multiple times and for fixed images that are at different fixed positions. Applicant has failed to address the indefinite language.
Examiner maintains the rejection.
Applicant's arguments filed 09/29/2026 have been fully considered but they are not persuasive.
Regarding the 112(b) rejections of the claims’ recitation of updating the composite image, Applicant argues “The Examiner questioned how the composite image is updated to include a virtual 3D image. Paragraphs 73-74 explicitly explain that the mixed reality display starts by overlaying a composite image (real-time ultrasound slice linked to the probe). When outlines extracted across the pinned fixed images are accumulated in real time into a 3D volume object, the composite display output in mixed reality is updated to incorporate this generated 3D volume object alongside the real-time scanning slice, enabling the clinician to view the accumulated 3D anatomical structure (e.g., blood vessels, muscles, lesions) in true 3D spatial alignment with the live probe and patient anatomy.”
Examiner disagrees. Again, Applicant is reminded that the claims are rejected under 112(b) and not 112(a). The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound. Neither Applicant nor the claim language address this issue of what the composition of the composite image is such that the 3D image is providing and addition to. The preceding element establishes that the composite image is an ultrasound image that is acquired.
Examiner maintains the rejection
Applicant’s arguments with respect to claims 1 and 13-14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1 and 13-16 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “wherein the extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. Paragraph 0074 discloses that “… such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects (e.g., blood vessels, muscles, fascia, organs, etc.) in a manner to connect the extracted points, accumulate the outlines in real time to be generated as a volume object, and update the generated volume object to the composite image…”. Paragraph 0075 discloses “…the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”. However, the specification fails to disclose the extraction being done by comparing the grayscale values for extraction based on differences to an extraction reference value. Rather, the specification disclose that it is according to a “certain level”. The specification does not disclose that the extraction reference value is the basis for the comparison with respect to differences. Instead, the specification discloses that the extraction reference value is for extraction of the plurality of fixed images based on user input and is adjustable. There is no disclosure of the comparison of the comparing grayscale of each pixel to extract points corresponding to differences that are set above an extraction threshold. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 13 recites “wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. Paragraph 0074 discloses that “… such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects (e.g., blood vessels, muscles, fascia, organs, etc.) in a manner to connect the extracted points, accumulate the outlines in real time to be generated as a volume object, and update the generated volume object to the composite image…”. Paragraph 0075 discloses “…the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”. However, the specification fails to disclose the extraction being done by comparing the grayscale values for extraction based on differences to an extraction reference value. Rather, the specification disclose that it is according to a “certain level”. The specification does not disclose that the extraction reference value is the basis for the comparison with respect to differences. Instead, the specification discloses that the extraction reference value is for extraction of the plurality of fixed images based on user input and is adjustable. There is no disclosure of the comparison of the comparing grayscale of each pixel to extract points corresponding to differences that are set above an extraction threshold. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 14 recites “wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. Paragraph 0074 discloses that “… such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects (e.g., blood vessels, muscles, fascia, organs, etc.) in a manner to connect the extracted points, accumulate the outlines in real time to be generated as a volume object, and update the generated volume object to the composite image…”. Paragraph 0075 discloses “…the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”. However, the specification fails to disclose the extraction being done by comparing the grayscale values for extraction based on differences to an extraction reference value. Rather, the specification disclose that it is according to a “certain level”. The specification does not disclose that the extraction reference value is the basis for the comparison with respect to differences. Instead, the specification discloses that the extraction reference value is for extraction of the plurality of fixed images based on user input and is adjustable. There is no disclosure of the comparison of the comparing grayscale of each pixel to extract points corresponding to differences that are set above an extraction threshold. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(a) are also rejected because they inherit the deficiencies of the claims they respectively depend upon.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 and 13-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 is indefinite for the following reasons:
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”2.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the input is. Preceding claim element establishes that the input is for the capture of a composite image. The instant element establishes the input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “wherein the extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what differences are being referred to. A grayscale image has pixels that make up the overall image based on brightness intensities3. If is further unclear if the claim is attempting to establish that the differential is thresholded or the grayscale value is thresholded.
Applicant is encouraged to provide consistent and clear language.
Recites “updating the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
Claim 13 is indefinite for the following reasons:
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”4.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the input is. Preceding claim element establishes that the input is for the capture of a composite image. The instant element establishes the input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what differences are being referred to. A grayscale image has pixels that make up the overall image based on brightness intensities5. If is further unclear if the claim is attempting to establish that the differential is thresholded or the grayscale value is thresholded.
Applicant is encouraged to provide consistent and clear language.
Recites “update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
Claim 14 is indefinite for the following reasons:
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”6.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the input is. Preceding claim element establishes that the input is for the capture of a composite image. The instant element establishes the input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what differences are being referred to. A grayscale image has pixels that make up the overall image based on brightness intensities7. If is further unclear if the claim is attempting to establish that the differential is thresholded or the grayscale value is thresholded.
Applicant is encouraged to provide consistent and clear language.
Recites “update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(b) are also rejected because they inherit the indefiniteness of the claims they respectively depend upon.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Fronk et al. (PGPUB No. US 2013/0267838) in view of Tolkowsky et al. (PGPUB No. US 2014/0094692) further in view of Wang et al. (PGPUB No. US 2022/0071596).
Regarding claim 1, Fronk teaches a method of displaying an ultrasound image based on mixed reality on a head-mounted display (HMD) (Abstract teaches ultrasound imaging and 3D modeling on an HMD), comprising:
obtaining the ultrasound image captured through an ultrasound device comprising a probe; obtaining localization information of the probe of the ultrasound probe (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe);
obtaining localization information of the probe of the ultrasound device (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
converting the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Abstract teaches the AR system);
outputting the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer);
receiving an input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
displaying a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a field of view of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system. A series of images are acquired at different points of time and are processed for vessel identification):
extracting three-dimensional (3D) boundary information on a captured object of the ultrasound device based on grayscale values of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times, wherein the extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
updating the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object(Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0035 teaches that after the 3D model is generated it can be viewed through the HMD and the vessel of interest can be observed with respect to the patient surface).
However, Fronk is silent regarding a method, wherein the HMD comprises an input device;
variably adjusting the extraction reference value based on an adjustment input applied through the input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches a method, wherein the HMD comprises an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified method would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
In an analogous imaging field of endeavor, regarding thresholding, Wang teaches a method, variably adjusting the extraction reference value based on an adjustment input applied through the input device (Claim 4 teaches a user input to adjust threshold levels. Paragraph 0030 teaches that thresholding can be used for segmentation).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk and Tolkowsky with Wang’s teaching of input a threshold. This modified method would allow the user to control and visualize the areas of interest (Abstract of Wang). Furthermore, the modification provides improved resolution and reduced artifacts (Paragraph 0003 of Wang).
Regarding claim 13, Fronk teaches an apparatus for providing ultrasound image display service based on mixed reality, comprising:
at least one processor (Fig. 1) configured to:
obtain an ultrasound image captured through an ultrasound device comprising a probe and to obtain localization information of the probe of the ultrasound device (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe. Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
convert the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
transmit the composite image and the localization information of the probe of the ultrasound device to a head-mounted display (HMD) via a network (See Fig. 1),
outputs the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer);
receive an input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
display a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a field of view of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system. A series of images are acquired at different points of time and are processed for vessel identification):
extract three-dimensional (3D) boundary information on a captured object of the ultrasound device based on grayscale values of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times, wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0035 teaches that after the 3D model is generated it can be viewed through the HMD and the vessel of interest can be observed with respect to the patient surface).
However, Fronk is silent regarding an apparatus, wherein the HMD comprises an input device;
variably adjust the extraction reference value based on an adjustment input applied through the input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches an apparatus, wherein the HMD comprises an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified apparatus would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
In an analogous imaging field of endeavor, regarding thresholding, Wang teaches an apparatus, variably adjust the extraction reference value based on an adjustment input applied through the input device (Claim 4 teaches a user input to adjust threshold levels. Paragraph 0030 teaches that thresholding can be used for segmentation).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk and Tolkowsky with Wang’s teaching of input a threshold. This modified apparatus would allow the user to control and visualize the areas of interest (Abstract of Wang). Furthermore, the modification provides improved resolution and reduced artifacts (Paragraph 0003 of Wang).
Regarding claim 14, Fronk teaches a system for displaying an ultrasound image based on mixed reality, comprising:
an ultrasound device comprising a probe to capture an ultrasound image (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe);
at least one processor (Fig. 1) to obtain the ultrasound image and localization information of the probe, and to convert the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Abstract teaches the AR system);
a head-mounted display (HMD) to receive the composite image and the localization information of the probe from the at least one processor and to output the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer), wherein the at least one processor (Fig. 1) is configured to:
receive an input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
display a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a field of view of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
extract three-dimensional (3D) boundary information on a captured object of the ultrasound device based on grayscale values of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the input applied multiple times, wherein extracting the 3D boundary information comprises comparing the grayscale values of the each pixel that constitutes the plurality of fixed images to extract points corresponding to differences above an extraction reference value, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume device (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ).
However, Fronk is silent regarding a system, a head-mounted display (HMD) comprising an input device;
variably adjust the extraction reference value based on an adjustment input applied through the input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches a system, a head-mounted display (HMD) comprising an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified apparatus would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
In an analogous imaging field of endeavor, regarding thresholding, Wang teaches a method, variably adjust the extraction reference value based on an adjustment input applied through the input device (Claim 4 teaches a user input to adjust threshold levels. Paragraph 0030 teaches that thresholding can be used for segmentation).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk and Tolkowsky with Wang’s teaching of input a threshold. This modified apparatus would allow the user to control and visualize the areas of interest (Abstract of Wang). Furthermore, the modification provides improved resolution and reduced artifacts (Paragraph 0003 of Wang).
Regarding claim 15, modified Fronk teaches the system in claim 14, as discussed above.
Fronk further teaches a system, wherein the localization information is measured and transmitted to the at least one processor (Paragraph 0027 teaches that the position and orientation of the transducer is tracked and recorded in the computer. See Fig. 1 and 4).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Fronk et al. (PGPUB No. US 2013/0267838) in view of Tolkowsky et al. (PGPUB No. US 2014/0094692) further in view of Wang et al. (PGPUB No. US 2022/0071596) further in view of Caluser et al. (PGPUB No. US 2015/0051489).
Regarding claim 16, modified Fronk teaches the system in claim 15, as discussed above.
Fronk further teaches a system, wherein the localization information comprises a position information, an angle information (Paragraph 0027 teaches that the position and orientation of the transducer is tracked and recorded in the computer. See Fig. 1 and 4. Paragraph 0031 teaches the tracking of the angle of the transducer).
However, the combination of Fronk, Tolkowsky, and Wang is silent regarding a system, wherein the localization information comprises a movement speed information, and a path information of a tracking device.
In an analogous imaging field of endeavor, regarding ultrasound probe tracking, Caluser teaches a system, wherein the localization information comprises a position information, an angle information, a movement speed information, and a path information of a tracking device (Paragraph 0112 teaches the
collection of speed, position, and orientation of the probe. Paragraph 0140 teaches that the speed,
position, and orientation are continuously monitoring. Paragraph 0137 teaches the assessment and display of the images across the anatomical region. Paragraph 0126 teaches the assessment of the viewing angle. Fig. 12 shows the scan path).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Fronk, Tolkowsky, and Wang with Caluser's teaching of localization information based on position, angle, speed, and path information. This modified apparatus would allow the user to improve image quality (Paragraph 0156 of Caluser). Furthermore, the modification will significantly reduce the time of the examination by eliminating the time-consuming manual labeling of images and speeding up the target finding at subsequent examinations (Paragraph 0013 of Caluser).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Gardi et al. (PGPUB No. US 2007/0270687): Teaches segmentation according to an adjustable threshold.
Min et al. (US Patent No. 12,193,862): Teaches segmentation according to an adjustable threshold.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADIL PARTAP S VIRK whose telephone number is (571)272-8569. The examiner can normally be reached Mon-Fri 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADIL PARTAP S VIRK/Primary Examiner, Art Unit 3798
1 Link: https://www.merriam-webster.com/dictionary/image
2 Link: https://www.merriam-webster.com/dictionary/image
3 “Often, the grayscale intensity is stored as an 8-bit integer giving 256 possible different shades of gray from black to white.” (Link: https://homepages.inf.ed.ac.uk/rbf/HIPR2/gryimage.htm)
4 Link: https://www.merriam-webster.com/dictionary/image
5 “Often, the grayscale intensity is stored as an 8-bit integer giving 256 possible different shades of gray from black to white.” (Link: https://homepages.inf.ed.ac.uk/rbf/HIPR2/gryimage.htm)
6 Link: https://www.merriam-webster.com/dictionary/image
7 “Often, the grayscale intensity is stored as an 8-bit integer giving 256 possible different shades of gray from black to white.” (Link: https://homepages.inf.ed.ac.uk/rbf/HIPR2/gryimage.htm)
