DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s statement on Page 7 regarding the certified copy of the priority document is noted. See below.
Applicant’s argument on Page 7 regarding the objection to the drawings has been fully considered. The objection to the drawings is withdrawn in view of the amendments.
Applicant’s argument on Page 7 regarding the objections to Claims 5 and 9 has been fully considered. The objections to Claims 5 and 9 are withdrawn in view of the amendments.
Applicant’s argument on Page 8 regarding the rejection of Claims 3-4 under 35 U.S.C. 112(b) has been fully considered. The rejection of Claims 3-4 under 35 U.S.C. 112(b) is withdrawn in view of the amendments.
Applicant’s argument on Pages 8-10 regarding the rejection of Claim 1 under 35 U.S.C. 103 over Oura in view of Ichiro has been fully considered but is not persuasive. Applicant argues that Ichiro “fails to teach or suggest the aforementioned claim features, especially ‘an attachment state of the camera with respect to the ultrasound probe.’” However, Ichiro states that the probe viewpoint camera 2 has a positional relationship with the ultrasonic probe 1 ([0028]), which is indicated by the position of the ultrasonic probe 1 that is specified using the numbers displayed in the probe moving image. Moreover, Ichiro further includes a panoramic camera 22 that is capable of capturing the relative positional relationship among the patient 11 and the ultrasonic probe 1 ([0029]-[0031]).
While differences between the prior art and the instant application are appreciated, they are not embodied in the claims in such a way as to differentiate.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-(d) prior to declaration of an interference, a certified English translation of the foreign application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e).
Failure to provide a certified translation may result in no benefit being accorded for the non-English application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 9-11, and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792) cited with paragraph numbers from the Espacenet translation.
Regarding Claim 1, Oura teaches an ultrasound diagnostic support device, (Abstract “ultrasonic diagnostic apparatus”), comprising:
a) a first hardware processor, ([0044] “ultrasonic image generation part 14”), that generates an ultrasound image based on an ultrasound signal from an ultrasound probe that transmits and receives ultrasound waves ([0050] “The ultrasonic image generation part 14 acquires a reception signal (acoustic line data) from the reception part 13, and generates an ultrasonic image (that is, a tomographic image) of the inside of the subject HT.”); and
b) a second hardware processor, ([0044] “optical image generation part 17”), that generates a camera image based on an imaging signal from a camera attached to the ultrasound probe (Fig. 1 and [0056] “The optical image generation part 17 acquires an image signal from the optical camera 30, and generates image data related to the optical image.”).
However, Oura does not explicitly teach a third hardware processor that stores camera attachment information indicating an attachment state of the camera with respect to the ultrasound probe in association with the ultrasound image and the camera image.
In an analogous ultrasonic diagnostics field of endeavor, Ichiro teaches an ultrasound diagnostic support device, ([0001] “ultrasonic diagnostic apparatus”), comprising: a third hardware processor that stores camera attachment information indicating an attachment state of the camera with respect to the ultrasound probe in association with the ultrasound image and the camera image (Figs. 3-4, [0028] “The probe moving image displayed in the probe moving image display area 8 is an image taken by the probe viewpoint camera 2 and represents two numbers for specifying the position of each area described on the probe position information sheet 21. The surgeon can specify the position of the probe viewpoint camera 2 by this number. Since the probe viewpoint camera 2 has a positional relationship with the ultrasonic probe 1, the position of the ultrasonic probe 1 can be specified using the numbers displayed in the probe moving image” and [0038] “The image superimposition processing unit 5b stores the created superimposed image data in the image recording / playback processing unit 10.” Where the camera attachment information is the numbers that specify the position of the probe viewpoint camera 2, which are part of the superimposed image data displayed, see Figs. 3-4, which is then stored by the image superimposition processing unit 5b (third hardware processor).).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify the teachings of Oura with Ichiro because it ensures accuracy in recognition at the point in which the ultrasound image is captured of the patient. This is advantageous in instances of surgical planning and diagnosis.
Regarding Claim 2, the modified device of Oura teaches all limitations of Claim 1, as discussed above. Furthermore, Ichiro teaches wherein the camera attachment information includes posture information indicating an attachment posture of the camera and position information indicating an attachment position of the camera with respect to the ultrasound probe ([0002] “The positional relationship between the patient 111 and the ultrasonic probe 101 is photographed by the panoramic camera 122” and [0003] “The superimposed image is composed of an image 108 and an ultrasonic image 107 taken by the panoramic camera 122.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify the teachings of Oura with Ichiro for the same reasons as Claim 1 above.
Regarding Claim 3, the modified device of Oura teaches all limitations of Claim 1, as discussed above. Furthermore, Ichiro teaches wherein the third hardware processor displays the ultrasound image, the camera image, and the camera attachment information associated with each other on split-screen display (Figs. 3-4).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify the teachings of Oura with Ichiro because the combined display minimizes distraction of an operator; there is no need to flip between multiple displays or windows, simplifying operation.
Regarding Claim 4, the modified device of Oura teaches all limitations of Claim 3, as discussed above. Furthermore, Ichiro teaches wherein the third hardware processor displays the camera attachment information in a superimposed manner on the camera image (Figs. 3-4 and [0003] “The image superimposing processing unit 105 creates superimposed image data in which an image captured by the panoramic camera 122 and an ultrasonic image detected by the ultrasonic probe 101 are superimposed based on the still image data and the ultrasonic image data.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify the teachings of Oura with Ichiro because the modification allows for the user to observe the image data in a single display, minimizing the need to pan through multiple windows or displays.
Regarding Claim 9, Oura teaches an ultrasonic diagnostic device, (Abstract “ultrasonic diagnostic apparatus”), comprising:
a) an ultrasound diagnostic device body to which the ultrasound diagnostic support device according to claim 1 is applied (Fig. 1 and as discussed above); and
b) an ultrasound probe ([0037] “ultrasonic probe 20”); and
c) a camera ([0038] “optical camera 30”).
Regarding Claim 10, Oura teaches all the limitations of Claim 9, as discussed above. Furthermore, Oura teaches wherein the camera changes an attachment posture and/or an attachment position with respect to the ultrasound probe ([0078] “the attachment 20T may be capable of adjusting orientations of the optical camera 30”).
Regarding Claim 11, Oura teaches all limitations of Claim 9, as discussed above. Furthermore, Oura teaches wherein the camera is attachable to and detachable from the ultrasound probe ([0078] “the camera 30 and the laser pointer 40 are attached to the housing 21 of the ultrasonic probe 20 via the detachable attachment 20T”).
Regarding Claims 13-14, Oura teaches an ultrasound diagnostic support method, (Fig. 7), comprising:
a) generating an ultrasound image based on an ultrasound signal from an ultrasound probe that transmits and receives ultrasound waves ([0050] “The ultrasonic image generation part 14 acquires a reception signal (acoustic line data) from the reception part 13, and generates an ultrasonic image (that is, a tomographic image) of the inside of the subject HT.”); and
b) generating a camera image based on an imaging signal from a camera attached to the ultrasound probe (Fig. 1 and [0056] “The optical image generation part 17 acquires an image signal from the optical camera 30, and generates image data related to the optical image.”).
Furthermore, the cited actions are computer implemented, which necessitate associated computer-readable media, as in [0061] (“The control part 19 includes a central processing unit (CPU) 191 as an arithmetic/control device, a read only memory (ROM) 192 and a random access memory (RAM) 193 as a main storage device, and the like. The ROM 192 stores a basic program and basic setting data. The CPU 191 reads a program corresponding to a processing content from the ROM 192, develops the program in the RAM 193, and executes the developed program, to centrally control operations of individual functional blocks (the operation input part 11, the transmission part 12, the reception part 13, the ultrasonic image generation part 14, the image composition part 15, the display part 16, the optical image generation part 17, and the oscillation control part 18) of the ultrasonic diagnostic apparatus main body 10.”).
However, Oura does not explicitly teach storing camera attachment information indicating an attachment state of the camera with respect to the ultrasound probe in association with the ultrasound image and the camera image.
In an analogous ultrasonic diagnostics field of endeavor, Ichiro teaches an ultrasound diagnostic support device, ([0001] “The present invention relates to an ultrasonic diagnostic apparatus capable of displaying an ultrasonic image and information for specifying a probe position”), comprising: storing camera attachment information indicating an attachment state of the camera with respect to the ultrasound probe in association with the ultrasound image and the camera image (Figs. 3-4, [0028] “The probe moving image displayed in the probe moving image display area 8 is an image taken by the probe viewpoint camera 2 and represents two numbers for specifying the position of each area described on the probe position information sheet 21. The surgeon can specify the position of the probe viewpoint camera 2 by this number. Since the probe viewpoint camera 2 has a positional relationship with the ultrasonic probe 1, the position of the ultrasonic probe 1 can be specified using the numbers displayed in the probe moving image” and [0038] “The image superimposition processing unit 5b stores the created superimposed image data in the image recording / playback processing unit 10.” Where the camera attachment information is the numbers that specify the position of the probe viewpoint camera 2, which are part of the superimposed image data displayed, see Figs. 3-4, which is then stored by the image superimposition processing unit 5b).
Furthermore, the cited actions are computer implemented, which necessitate associated computer-readable media, as in [0037]-[0038] (“imaging superimposition processing unit 5b”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify the teachings of Oura with Ichiro because it ensures accuracy in recognition at the point in which the ultrasound image is captured of the patient. This is advantageous in instances of surgical planning and diagnosis.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792), cited with paragraph numbers from the Espacenet translation as applied to Claim 1 above, and further in view of Toporek et al. (US 20220092800).
Regarding Claim 5, the modified device of Oura teaches all limitations of Claim 1, as discussed above. However, the modified device of Oura does not explicitly teach wherein a fourth hardware processor that sets the camera attachment information.
In an analogous real-time tracking of ultrasound imagery field of endeavor, Toporek teaches an ultrasound diagnostic support device further comprising a fourth hardware processor that sets the camera attachment information ([0093] “the camera system 140 […] controlled by the robotic system to maintain a view of the hybrid marker 110,” where the camera system being controlled by the robotic system to maintain a view of the hybrid marker is interpreted as camera attachment information because the camera system must be attached in a certain manner in order to maintain a view of the hybrid marker.).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the teachings of Toporek because the modification allows for customization of tracking of the intended objects (needles, markers, etc.) within the frame of view of the camera.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792), cited with paragraph numbers from the Espacenet translation, and Toporek et al. (US 20220092800), as applied to Claim 5 above, and further in view of Pujari et al. (US 20230125813).
Regarding Claim 6, the modified device of Oura teaches all limitations of Claim 5, as discussed above. However, the modified device of Oura does not explicitly teach wherein the fourth hardware processor presents an icon indicating the attachment state of the camera with respect to the ultrasound probe, and sets the camera attachment information based on an icon operation using the icon.
In an analogous wireless link quality of handheld ultrasound devices field of endeavor, Pujari teaches an ultrasonic diagnostic support device, ([0002] “The disclosure relates to a wireless ultrasound probe, an ultrasound diagnostic apparatus connected to the wireless ultrasound probe”), wherein the fourth hardware processor presents an icon indicating the attachment state of the camera with respect to the ultrasound probe, ([0038] “an exemplary ultrasound image display 200 having an indication 202 of a good connection status”), and sets the camera attachment information based on an icon operation using the icon (Fig. 4, where the connections state determines the flow of operations of the probe. While Song does not explicitly teach a camera, the attached camera is taught by the modified device of Oura, as above. Because the camera is attached to the ultrasound probe (i.e., the attachment/connection information would be the same), the probe connection status of Song is interpreted as the camera attachment information.).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the teachings of Pujari because the modification ensures reliable imaging data from the device, as taught by Pujari in [0003].
Regarding Claim 7, the modified device of Oura teaches all limitations of Claim 5, as discussed above. Furthermore, Pujari teaches wherein the fourth hardware processor presents a plurality of options indicating the attachment state of the camera with respect to the ultrasound probe, ([0029] “The user input device 130 may be utilized to input […] imaging modes, image acquisition and scan parameters, settings, configuration parameters, select protocols and/or templates, select wireless connection state thresholds”), and sets the camera attachment information based on an operation of selecting one of the plurality of options ([0029] “the user input device 130 may be operable to configure, manage and/or control operation of one or more components and/or modules in the ultrasound system 100. In this regard, the user input device 130 may be operable to configure, manage and/or control operation of the transmitter 102, the ultrasound probe 104, the transmit beamformer 110, the receiver 118, the receive beamformer 120, the RF processor 124, the RF/IQ buffer 126, the user input device 130, the probe signal processor 132, the image buffer 136, the display system 134, the archive 138, and/or the probe and display device transceivers.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the teachings of Pujari because the selection of the connection states by the user allow the user to have some control over the quality of the imaging procedure, some procedures may need higher quality imaging than others.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792), cited with paragraph numbers from the Espacenet translation, and Toporek et al. (US 20220092800), as applied to Claim 5 above, and further in view of Mihailescu et al. (US 20160242744)
Regarding Claim 8, the modified device of Oura teaches all limitations of Claim 5, as discussed above. However, the modified device of Oura does not explicitly teach wherein the fourth hardware processor acquires the camera image captured by the camera in a state where the ultrasound probe is installed at a predetermined position of a calibration sheet including identification information for identifying an attachment state of the camera with respect to the ultrasound probe, and wherein the fourth hardware processor sets the camera attachment information based on the identification information included in the camera image.
In an analogous tracking and guiding sensors and instruments field of endeavor, Mihailescu teaches an ultrasonic diagnostic support device, (Abstract “A shared-housing ultrasound transducer and machine-vision camera system is disclosed”),
a) wherein the fourth hardware processor, ([0081] “computing unit 104”), acquires the camera image, ([0169] “Mechanically registered to the probe is a camera housing shell 1004 comprising a camera whose lenses 1005 are oriented in the general direction of the patient.”), captured by the camera in a state where the ultrasound probe is installed at a predetermined position of a calibration sheet including identification information for identifying an attachment state of the camera with respect to the ultrasound probe, ([0171] “Fiduciary elements can be placed on the patient or on stable objects adjacent to the patient to give a reference frame for the virtual reality model and to provide the proper scale for the whole environment when using the computer vision system for registering the ultrasound probe into the 3-D model. The fiduciary element can be made of a patterned layer made of various colors or shades, can comprise reflective objects,” Claim 1 “cause the at least one processor to determine a pose of the camera with respect to the fiducial marker using an image of the fiducial marker captured by the camera,” and Claim 2 “the fiducial marker includes binary coding”), and
b) wherein the fourth hardware processor sets the camera attachment information based on the identification information included in the camera image ([0116] “The second computer vision analysis step 602 identifies the fiducial object or fiducial marker in the field of view, and uses its apparent shape to determine the position and orientation of the computer vision camera in respect to that fiducial object in the following pose estimator step 603.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the teachings of Mihailescu because the modification ensures that the imaging procedure is as accurate as intended by the operator; the predetermined position of a calibration sheet including identification information for identifying an attachment state of the camera with respect to the ultrasound probe, which is interpreted as the placed fiducial markers that determine the positioning of the probe and camera of Mihailescu, allow for the operator to accurately image the subject as intended.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792), cited with paragraph numbers from the Espacenet translation as applied to Claim 9 above, and further in view of Miyachi (WO 2022202289), cited with paragraph numbers from its respective US Patent Application Publication containing the same information, US 20240000439).
Regarding Claim 12, the modified device of Oura teaches all limitations of Claim 9, as discussed above. However, the modified device of Oura does not explicitly teach wherein the camera includes a wide-angle lens.
In an analogous ultrasound diagnostics field of endeavor, Miyachi teaches an ultrasound diagnostic device, (Abstract “ultrasound diagnostic apparatus”), wherein the camera includes a wide-angle lens ([0171] “In addition, the apparatus main body 3C comprises […] a second optical camera 67” and [0176] “second optical camera 67 is a wide-angle camera”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the teachings of Miyachi because the camera offers a wider view of the region of interest than that of a standard angle lens.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Oura et al. (US 20230263575) in view of Ichiro (JP 2007282792), cited with paragraph numbers from the Espacenet translation, and Toporek et al. (US 20220092800), as applied to Claim 1 above, and further in view of Wang et al. (CN 114886459), cited with page numbers from the Espacenet translation.
Regarding Claim 15, the modified device of Oura teaches all limitations of Claim 1, as discussed above. However, the modified device of Oura does not explicitly teach wherein the camera attachment information includes position information indicating an attachment position of the camera with respect to the ultrasound probe or posture information indicating an attachment posture of the camera with respect to the ultrasound probe.
In an analogous ultrasonic operation field of endeavor, Wang teaches an ultrasound diagnostic device, (Abstract “the handheld device comprises an ultrasonic probe”), wherein the camera attachment information includes position information indicating an attachment position of the camera with respect to the ultrasound probe or posture information indicating an attachment posture of the camera with respect to the ultrasound probe (Page 6 “The pose data of the ultrasound probe 121 at this time can be obtained according to the relative pose relationship between the camera 122 and the ultrasound probe 121 and the pose data when the camera 122 collects the identifiable code image. In this way, as the doctor holds the handheld device 12 and uses the ultrasound probe 121 to collect the ultrasound image of the patient, the camera 122 on the handheld device 12 will collect the identifiable code image, and the data host 13 can obtain the ultrasound probe 121 to collect the ultrasound accordingly.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to further modify with the camera attachment information of Wang because the modification assists an operator in navigating the imaging process with proper imaging technique and positioning.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIA CHRISTINA TALTY whose telephone number is (571)272-8022. The examiner can normally be reached M-Th 8:30-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mike Carey can be reached at (571) 270-7235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARIA CHRISTINA TALTY/Examiner, Art Unit 3797
/MICHAEL J CAREY/Supervisory Patent Examiner, Art Unit 3795
Prosecution Insights