DETAILED ACTION
This office action is in response to the communication received on 06/11/2026 concerning application no. 18/703,422 filed on 04/22/2024.
Claims 1 and 13-16 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1 and 13-14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 14 objected to because of the following informalities:
Claim 14 recites “a a composite image”. This claim element should be amended to “a composite image”. This amendment will proper grammar.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1 and 13-16 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 is rejected under 35 U.S.C. 112(a) for the following reasons:
Recites “based on color information including a gray scale”. While paragraph 0074 discloses “compare color information, such as gray scale, of each pixel”, the specification fails to disclose the color information in relation to the gray scale. The specification fails to disclose the manner of operation for the gray scale and color information to be able to operate together for the claimed function. The specification fails to establish consideration, disclosure of the manner of incorporation of the gray scale into the color information, and the manner of use of those two forms of dataset together. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level”. While paragraph 0074 discloses the “compare color information, such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects”, the specification fails to disclose the comparison of a gray scale difference. The specification fails to disclose the determination or usage of a gray scale difference. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “a certain level”. While the paragraph 0074 discloses “a certain level” being used, the specification fails to disclose what that certain level is or what manner that level is ascertained. There is no disclosure on how the certain value is ascertained or implemented in the claimed invention. The specification does not disclose what “certainty” is for the cited value. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “adjusting an extraction reference value”. While the paragraph 0075 discloses “More specifically, the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”, the specification fails to disclose what that extraction value is and in what manner it is implemented and used to extract the plurality of fixed images. The specification does not disclose the manner of operation, what the value is or what it is representative of, and how it performs the claimed extraction. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 13 is rejected under 35 U.S.C. 112(a) for the following reasons:
Recites “based on color information including a gray scale”. While paragraph 0074 discloses “compare color information, such as gray scale, of each pixel”, the specification fails to disclose the color information in relation to the gray scale. The specification fails to disclose the manner of operation for the gray scale and color information to be able to operate together for the claimed function. The specification fails to establish consideration, disclosure of the manner of incorporation of the gray scale into the color information, and the manner of use of those two forms of dataset together. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level”. While paragraph 0074 discloses the “compare color information, such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects”, the specification fails to disclose the comparison of a gray scale difference. The specification fails to disclose the determination or usage of a gray scale difference. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “a certain level”. While the paragraph 0074 discloses “a certain level” being used, the specification fails to disclose what that certain level is or what manner that level is ascertained. There is no disclosure on how the certain value is ascertained or implemented in the claimed invention. The specification does not disclose what “certainty” is for the cited value. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “adjusting an extraction reference value”. While the paragraph 0075 discloses “More specifically, the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”, the specification fails to disclose what that extraction value is and in what manner it is implemented and used to extract the plurality of fixed images. The specification does not disclose the manner of operation, what the value is or what it is representative of, and how it performs the claimed extraction. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claim 14 is rejected under 35 U.S.C. 112(a) for the following reasons:
Recites “based on color information including a gray scale”. While paragraph 0074 discloses “compare color information, such as gray scale, of each pixel”, the specification fails to disclose the color information in relation to the gray scale. The specification fails to disclose the manner of operation for the gray scale and color information to be able to operate together for the claimed function. The specification fails to establish consideration, disclosure of the manner of incorporation of the gray scale into the color information, and the manner of use of those two forms of dataset together. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level”. While paragraph 0074 discloses the “compare color information, such as gray scale, of each pixel that constitutes the fixed images to extract points corresponding to differences above a certain level and then extract outlines of the captured objects”, the specification fails to disclose the comparison of a gray scale difference. The specification fails to disclose the determination or usage of a gray scale difference. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “a certain level”. While the paragraph 0074 discloses “a certain level” being used, the specification fails to disclose what that certain level is or what manner that level is ascertained. There is no disclosure on how the certain value is ascertained or implemented in the claimed invention. The specification does not disclose what “certainty” is for the cited value. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Recites “adjusting an extraction reference value”. While the paragraph 0075 discloses “More specifically, the service providing device 100 may variably adjust an extraction reference value (e.g., gray scale value, etc.) for extracting the plurality of fixed images based on the user's adjustment input (e.g., gesture-based slide adjustment, etc.) applied through the interaction device 500”, the specification fails to disclose what that extraction value is and in what manner it is implemented and used to extract the plurality of fixed images. The specification does not disclose the manner of operation, what the value is or what it is representative of, and how it performs the claimed extraction. Therefore, the claim contains subject matter which is not described in the specification in such a way as to reasonably convey to one with ordinary skill in the art that the inventor had possession of the claim invention at the time of filing.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(a) are also rejected because they inherit the deficiencies of the claims they respectively depend upon.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 and 13-16 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 is indefinite for the following reasons:
Recites “view of field”. This claim element is indefinite. It would be unclear what field is being referred to. The element is not a term of art and it would be unclear what view the claim is attempting to establish. It would be further unclear if the claim is attempting to establish a field of view.
Applicant is encouraged to provide consistent and clear language.
Recites “based on color information including a gray scale of each pixel”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the manner of operation of the system is in color or in gray scale. It is well known in the art that “Color images represent visual information in three channels—Red, Green, and Blue (RGB). Each pixel consists of three intensity values corresponding to these colors.”1 “Grayscale images contain shades of gray, ranging from black to white. Each pixel represents intensity information, with values typically ranging from 0 (black) to 255 (white) in an 8-bit image.”2
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”3.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the user input is. Preceding claim element establishes that the user input is for the capture of a composite image. The instant element establishes the user input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “extracting points corresponding to gray scale differences”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art in what manner a point is representative of a difference in the gray scale context. That is, a difference requires a variation while a point is a singular element. It is further unclear that the difference is with respect to.
Applicant is encouraged to provide consistent and clear language.
Recites “a certain level”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the claim is establishing a preset threshold value. Another interpretation is that the level is a user defined level.
Applicant is encouraged to provide consistent and clear language.
Recites “the gray scale”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what gray scale is being referred to. The prior element attempts to establish a gray scale for each pixel of each of a plurality of fixed images.
Applicant is encouraged to provide consistent and clear language.
Recites “extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “extraction reference value” is the same as the “certain level” established in the preceding claim element or is a separate and distinct feature. The certain level is defined to be for the extraction of the gray scale differences in the modeling process
Applicant is encouraged to provide consistent and clear language.
Recites “update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
The claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Claim 13 is indefinite for the following reasons:
Recites “view of field”. This claim element is indefinite. It would be unclear what field is being referred to. The element is not a term of art and it would be unclear what view the claim is attempting to establish. It would be further unclear if the claim is attempting to establish a field of view.
Applicant is encouraged to provide consistent and clear language.
Recites “based on color information including a gray scale of each pixel”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the manner of operation of the system is in color or in gray scale. It is well known in the art that “Color images represent visual information in three channels—Red, Green, and Blue (RGB). Each pixel consists of three intensity values corresponding to these colors.”4 “Grayscale images contain shades of gray, ranging from black to white. Each pixel represents intensity information, with values typically ranging from 0 (black) to 255 (white) in an 8-bit image.”5
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”6.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the user input is. Preceding claim element establishes that the user input is for the capture of a composite image. The instant element establishes the user input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “extracting points corresponding to gray scale differences”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art in what manner a point is representative of a difference in the gray scale context. That is, a difference requires a variation while a point is a singular element. It is further unclear that the difference is with respect to.
Applicant is encouraged to provide consistent and clear language.
Recites “a certain level”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the claim is establishing a preset threshold value. Another interpretation is that the level is a user defined level.
Applicant is encouraged to provide consistent and clear language.
Recites “the gray scale”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what gray scale is being referred to. The prior element attempts to establish a gray scale for each pixel of each of a plurality of fixed images.
Applicant is encouraged to provide consistent and clear language.
Recites “extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “extraction reference value” is the same as the “certain level” established in the preceding claim element or is a separate and distinct feature. The certain level is defined to be for the extraction of the gray scale differences in the modeling process
Applicant is encouraged to provide consistent and clear language.
Recites “update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
The claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Claim 14 is indefinite for the following reasons:
Recites “view of field”. This claim element is indefinite. It would be unclear what field is being referred to. The element is not a term of art and it would be unclear what view the claim is attempting to establish. It would be further unclear if the claim is attempting to establish a field of view.
Applicant is encouraged to provide consistent and clear language.
Recites “based on color information including a gray scale of each pixel”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the manner of operation of the system is in color or in gray scale. It is well known in the art that “Color images represent visual information in three channels—Red, Green, and Blue (RGB). Each pixel consists of three intensity values corresponding to these colors.”7 “Grayscale images contain shades of gray, ranging from black to white. Each pixel represents intensity information, with values typically ranging from 0 (black) to 255 (white) in an 8-bit image.”8
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the relationship of the plurality of fixed images is with respect to the preceding fixed image. The initial element establishes that a fixed image is displayed in which the composite image is continuously maintained.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions”. This claim element is indefinite. It would be unclear in what manner each of the fixed images is displayed in a plurality of different fixed positions. An image is a “a visual representation of something”9.
Applicant is encouraged to provide consistent and clear language.
Recites “each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what the user input is. Preceding claim element establishes that the user input is for the capture of a composite image. The instant element establishes the user input occurring multiple times and for fixed images that are at different fixed positions.
Applicant is encouraged to provide consistent and clear language.
Recites “extracting points corresponding to gray scale differences”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art in what manner a point is representative of a difference in the gray scale context. That is, a difference requires a variation while a point is a singular element. It is further unclear that the difference is with respect to.
Applicant is encouraged to provide consistent and clear language.
Recites “a certain level”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the claim is establishing a preset threshold value. Another interpretation is that the level is a user defined level.
Applicant is encouraged to provide consistent and clear language.
Recites “the gray scale”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art what gray scale is being referred to. The prior element attempts to establish a gray scale for each pixel of each of a plurality of fixed images.
Applicant is encouraged to provide consistent and clear language.
Recites “extraction reference value”. This claim element is indefinite. It would be unclear to one with ordinary skill in the art if the “extraction reference value” is the same as the “certain level” established in the preceding claim element or is a separate and distinct feature. The certain level is defined to be for the extraction of the gray scale differences in the modeling process
Applicant is encouraged to provide consistent and clear language.
Recites “update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object”. This claim element is indefinite. The preceding claim element establishes that the composite image is according to the ultrasound acquisition. It would be unclear to one with ordinary skill in the art what composes the composite image that the 3D image is provided in addition to. That is, the 3D image itself is the volume object of acquired features by ultrasound.
Applicant is encouraged to provide consistent and clear language.
The claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(b) are also rejected because they inherit the indefiniteness of the claims they respectively depend upon.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Fronk et al. (PGPUB No. US 2013/0267838) in view of Tolkowsky et al. (PGPUB No. US 2014/0094692).
Regarding claim 1, Fronk teaches a method of displaying an ultrasound image based on mixed reality on a head-mounted display (HMD) (Abstract teaches ultrasound imaging and 3D modeling on an HMD), comprising:
obtaining the ultrasound image captured through an ultrasound device comprising a probe (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe);
obtaining localization information of the probe of the ultrasound device (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
converting the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Abstract teaches the AR system);
outputting the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer);
receiving user input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
displaying a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the user input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a view of field of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system):
extracting three-dimensional (3D) boundary information on a captured object of the ultrasound device based on color information including a gray scale of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times, wherein extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level among the plurality of fixed images, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
updating the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object; and variably adjusting an extraction reference value of the gray scale for extracting the plurality of fixed images based on an adjustment input applied through the input device (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ).
However, Fronk is silent regarding a method,
wherein the HMD comprises an input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches a method,
wherein the HMD comprises an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified method would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
Regarding claim 13, Fronk teaches an apparatus for providing ultrasound image display service based on mixed reality, comprising:
at least one processor (Fig. 1) configured to:
obtain an ultrasound image captured through an ultrasound device comprising a probe and to obtain localization information of the probe of the ultrasound device (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe. Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
convert the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker);
transmit the composite image and the localization information of the probe of the ultrasound device to a head-mounted display (HMD) via a network (See Fig. 1),
outputs the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer);
receive user input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
display a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the user input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a view of field of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system):
extract three-dimensional (3D) boundary information on a captured object of the ultrasound device based on color information including a gray scale of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times, wherein extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level among the plurality of fixed images, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object; and variably adjust an extraction reference value of the gray scale for extracting the plurality of fixed images based on an adjustment input applied through the input device (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ).
However, Fronk is silent regarding an apparatus, wherein the HMD comprises an input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches an apparatus, wherein the HMD comprises an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified apparatus would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
Regarding claim 14, Fronk teaches a system for displaying an ultrasound image based on mixed reality, comprising:
an ultrasound device comprising a probe to capture an ultrasound image (Paragraph 0005-06 teaches ultrasound imaging with an ultrasound probe);
at least one processor (Fig. 1) to obtain the ultrasound image and localization information of the probe, and to convert the ultrasound image into a composite image to be output based on the mixed reality (Paragraph 0031 teaches that the position and orientation of the transducer is tracked via the marker. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Abstract teaches the AR system);
a head-mounted display (HMD) to receive the composite image and the localization information of the probe from the at least one processor and to output the composite image by overlaying the composite image on a reference image captured to include the probe (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0023 teaches the tracking of the markers and the association of the AR with respect to the real world. Paragraph 227 teaches that the marking includes the tracking of the transducer), wherein the at least one processor (Fig. 1) is configured to:
receive user input from the input device for capturing the composite image (Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system);
display a fixed image in which the composite image is continuously maintained at a predetermined fixed position on the reference image based on the user input, wherein a predetermined transparency is applied to the fixed image so as not to obstruct a view of field of a user (Paragraph 0022 teaches that the displays and the image is at least-semi transparent. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ. Paragraph 0020 teaches that the virtual image of the 3D model is superimposed on the clinician’s view for the HMD display. Paragraph 0028 teaches that the user can input the data and performance of the system):
extract three-dimensional (3D) boundary information on a captured object of the ultrasound device based on color information including a gray scale of each pixel constituting each of a plurality of fixed images in a state where the fixed image is individually displayed at a plurality of different fixed positions based on the user input applied multiple times, wherein extracting the 3D boundary information comprises extracting points corresponding to gray scale differences above a certain level among the plurality of fixed images, extracting outlines of the captured object by connecting the extracted points, and accumulating the outlines in real time to generate a volume object (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ);
update the composite image to include a virtual 3D image corresponding to the captured object based on the extracted 3D boundary information, wherein the virtual 3D image comprises the generated volume object; and variably adjust an extraction reference value of the gray scale for extracting the plurality of fixed images based on an adjustment input applied through the input device (Paragraph 0028 teaches the imaging of the vessel according to the user input. The images are captured at differing points of time and processed to identify the vessel. This can be done with grey-scale values that are thresholded. The pixels above the threshold are acquired and used for the indication of the vessel. Paragraph 0030 teaches that with perimeter positions identified in each of the filtered ultrasound images, a 3D model is generated and is connected between the perimeter positions for each of the images via interpolation. Paragraph 0046 teaches imaging in real time and the assessment of the structures in situ).
However, Fronk is silent regarding a system, a head-mounted display (HMD) comprising an input device.
In an analogous imaging field of endeavor, regarding ultrasound probe imaging and AR integration, Tolkowsky teaches a system, a head-mounted display (HMD) comprising an input device (Paragraph 0324 teaches that the input control can be via an input interface and it can be an HMD).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Fronk with Tolkowsky’s teaching of the integration of the input device into the HMD. This modified apparatus would allow the user to select and control the image selection as per the user’s desire (Paragraph 0326 of Tolkowsky). Furthermore, the modification allows for precise locational imaging of the regions of interest (Paragraph 0406 of Tolkowsky).
Regarding claim 15, modified Fronk teaches the system in claim 14, as discussed above.
Fronk further teaches a system, wherein the localization information is measured and transmitted to the at least one processor (Paragraph 0027 teaches that the position and orientation of the transducer is tracked and recorded in the computer. See Fig. 1 and 4).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Fronk et al. (PGPUB No. US 2013/0267838) in view of Tolkowsky et al. (PGPUB No. US 2014/0094692) further in view of Caluser et al. (PGPUB No. US 2015/0051489).
Regarding claim 16, modified Fronk teaches the system in claim 15, as discussed above.
Fronk further teaches a system, wherein the localization information comprises a position information, an angle information (Paragraph 0027 teaches that the position and orientation of the transducer is tracked and recorded in the computer. See Fig. 1 and 4. Paragraph 0031 teaches the tracking of the angle of the transducer).
However, the combination of Fronk and Tolkowsky is silent regarding a system, wherein the localization information comprises a movement speed information, and a path information of a tracking device.
In an analogous imaging field of endeavor, regarding ultrasound probe tracking, Caluser teaches a system, wherein the localization information comprises a position information, an angle information, a movement speed information, and a path information of a tracking device (Paragraph 0112 teaches the
collection of speed, position, and orientation of the probe. Paragraph 0140 teaches that the speed,
position, and orientation are continuously monitoring. Paragraph 0137 teaches the assessment and display of the images across the anatomical region. Paragraph 0126 teaches the assessment of the viewing angle. Fig. 12 shows the scan path).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the combination of Fronk and Tolkowsky with Caluser's teaching of localization information based on position, angle, speed, and path information. This modified apparatus would allow the user to improve image quality (Paragraph 0156 of Caluser). Furthermore, the modification will significantly reduce the time of the examination by eliminating the time-consuming manual labeling of images and speeding up the target finding at subsequent examinations (Paragraph 0013 of Caluser).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kano (PGPUB No. US 20240237969): Teaches input on HMD device of ultrasound imaging and the implementation of an AR with the display of the probe.
Mullick et al. (PGPUB No. US 2015/0148677): Teaches thresholding for segmentation.
Sui et al. (PGPUB No. US 2013/0046168): Teaches thresholding for segmentation.
Jin et al. (PGPUB No. US 2021/0128265): Teaches input on HMD device of ultrasound imaging and the implementation of an AR with the display of the probe.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADIL PARTAP S VIRK whose telephone number is (571)272-8569. The examiner can normally be reached Mon-Fri 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADIL PARTAP S VIRK/Primary Examiner, Art Unit 3798
1 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
2 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
3 Link: https://www.merriam-webster.com/dictionary/image
4 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
5 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
6 Link: https://www.merriam-webster.com/dictionary/image
7 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
8 Link: https://www.geeksforgeeks.org/computer-vision/how-do-you-decide-whether-to-utilize-grayscale-or-colour-images-as-input-for-computer-vision-tasks/
9 Link: https://www.merriam-webster.com/dictionary/image
Prosecution Insights