DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1, 3-14, 16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-12 are rejected under 35 U.S.C. 103 as being unpatentable over Shastri (US 7,259,729 B2) in view of Kohle (US 20180293773 A1; hereafter Kohle ), and futher in view of Zhao et al. (US 20230147826 A1; hereafter Zhao).
As of Claim 1: Shastri teaches a method for visualizing image data on a display device (5) of a medical device visualization system (1) (Col. 4, lines 27-40 and note that FIG. 1, this medical image network and as shown in FIG. 2, workstation 2 is provided with a controller 27 and a memory 26. Note that in the current embodiment of the two monitors, 22A and 22B are used for displaying images.) comprising a microscope, an exoscope (Col. 4, lines 40-55), the visualization system (1) comprising; at least one of a camera (2) or image sensor, which generates image data to be displayed on the display device (5),wherein various sequences of visualization modes are stored in a memory of a computer device (3) of the visualization system (1) (Col. 5, lines 23-36 and note that image display method for displaying images on monitor 22 of workstation 2 will be explained. FIG. 2 is a block diagram showing the detailed configuration of a workstation. As shown in FIG. 2, workstation 2 is provided with a controller 27 and a memory 26. Note that in the current embodiment of the two monitors, 22A and 22B are used for displaying images. In addition, controller 27 and memory 26 are provided in FIG. 2 for explanation purposes, however, actual processing is carried out by use of software installed in computer 21. In memory 26, multiple display protocol sequences PS, in which multiple display protocols P defining multiple display layouts are lined up in a predetermined order, are remembered.), and wherein the different visualization modes of a sequence each comprise a combination made up of: a respective imaging mode, which is based on a specific image signal processing of the received live image data, and an associated display layout, in which the live image data are presented on the display device, and the method comprises the following steps: individually selecting by a user, a predefined sequence (a-g) of visualization modes from the stored various sequences of visualization modes, the predefined sequence (a-) comprising a first and a second visualization mode, receiving (110) the image data; displaying (120) the received image data by the display device (5) in the first visualization mode of the predefined sequence (a-g) of visualization modes (Col. 6, lines 56-67 and note that note that for such display protocol sequences PS, there is a selection screen displayed on monitors 22A and 22B as shown In FIG. 4, from which it is possible to select by clicking with the mouse, a desired display protocol sequence PS. Further, at this time, it is beneficial that the display protocol sequence be selected corresponding to the examination, the series, the type of input modality 4 in which the images were formed (CR images, CT images, MRI images, etc.), the examining doctor or group of doctors to which the examining doctor belongs (reading doctor, clinical doctor, referring doctor, etc.) the place where the images are to be employed (an examination room, a doctor's residence, etc.), the purpose for which the images are to be used, (reading, reference, comparison, etc.), etc.); wherein the second visualization mode differs from the first visualization mode in at least one of the respective imaging mode or the associated display layout
Kohle is a similar or analogous system to the claimed invention as evidenced Kohle teaches would have prompted a predictable variation of Shastri by applying Kohle’s known principal of wherein the first visualization mode is based on a first parameter set of a multidimensional parameter space of visualization parameters (¶¶0027,0037,0039,0057-0064 and note also layout parameters are specified. The layout parameters stipulate in which way the image data set is displayed on a display. The layout parameters comprise, for example, the number of the image segments on the screen (2×2 or 3×3), an allocation of particular image series to the segments, a sequence of several layouts, a representation of image data within an image series (bone window, soft tissue window), zoom factors, synchronization settings or particular angles of view for a three-dimensional volume. The same applies for a plurality of studies on comparison with previous examinations. In general, the layout parameters can comprise all the parameters with which a particular display of the image data set 103 is controlled.) ; in reaction to receiving (130) a first user input of at least one user input, determining (140) a second parameter set of the multidimensional parameter space for the second visualization mode of the predefined sequence (a-g) of different visualization modes; and changing (150) from the first visualization mode to the second visualization mode based on the determined second parameter set (¶¶0057-0064).
In view of the motivations such increasing the efficiency in the findings, reduces diagnostic errors and supports standardization as thereby further improving initial input for the self-learning system in order initially to learn which parameters lead to which image impression one of ordinary skill in the art would have implemented the claimed variation of the prior art system of Shastri.
Therefore, the claimed invention would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Zhao is a similar or analogous system to the claimed invention as evidenced Zhao teaches improving surgeon's view and context awareness during laparoscopic and video assisted surgeries (Abstract) that would have prompted a predictable variation of Shastri by applying Zhao’s known principal of live image data; (¶0034 and note that screen layout 600, the left side may show CT, MRI, PET, ultrasound images and/or 3D model(s) 610 (e.g., in reconstructed 3D model), 612 (e.g., in CT axial view), 614 (e.g., in CT sagittal view), 616 (e.g., in CT coronal view) stored in storage 414, and the right side may show live laparoscopic images 620 from laparoscope 360), an endoscope (¶0044 of Zhao and note video-assisted thoracoscopic, endoscopic, percutaneous, colonoscopy surgeries, etc. Some or all of the disclosure relating to laparoscopes and laparoscopy may be similarly applied to other video-assisted surgical instruments and operations, such as endoscope and endoscopy, percutaneous scope(s) and surgery, colonoscope and colonoscopy, etc.).
In view of the motivations such as providing a user surgeon has sufficient knowledge to be able to label matching points or features between the left-side (e.g., static and/or live dynamic) content and the right-side (e.g., live dynamic) content shown in screen layout 600 thereby further improving surgeon's view during surgery, by utilizing surgeon's guidance sporadically to achieve high image registration accuracy of ¶0034 of Zhao that one of ordinary skill in the art would have implemented the claimed variation of the prior art system of Shastri.
Therefore, the claimed invention would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
As of Claim 3: Shastri in view of Kohle in view of Zhao further teaches the selected sequence of visualization modes comprises at least one of different imaging modes based on at least one of: a) different processing or filtering of the received live image data or b) recording by different spectral illumination from a light source (6),or (ii) different display layouts including at least one of: a full image layout, a split-screen layout, a picture-in-picture layout, a superposition layout, a picture-out-of-picture layout, a zoom layout, or a detail enlargement layout (Shastri Col. 5, lines 37-57 and note that FIGS. 3A-C show the display layouts according to each type of display protocol. Note that in the current embodiment, images obtained by a CT apparatus are displayed. More specifically, in a CT apparatus, two images are taken of the same position: one of the images has shadowed structures, and the other image does not. Generally, there is a set of projection images (called "scout views") composed of a projection image having shadowing and a projection image not having shadowing, and a set of cross-sectional images composed of a cross-sectional image having shadowing and a cross-sectional image not having shadowing, for a total of two sets of images. Each of the two sets of images obtained in this way are grouped so that there is a group containing an image S1 (projection image with shadowing), and an image S2 (cross-sectional image with shadowing), and a group containing an image S3 (projection image without shadowing), and an image S4 (cross-sectional image without shadowing). Here, there is only one image for each projection image; however, each cross-sectional image has 36 images, corresponding to the slice position, which are represented by S2 and S4. Note that images S1-S4 represent images obtained in the current examination. The current examination is interpreted as live).
As of Claim 4: Shastri in view of Kohle in view of Zhao further teaches at least one of:the second visualization mode directly precedes or directly follows the first visualization mode in the predefined sequence, or the change from the first visualization mode to the second visualization mode or to multiple second visualization modes of the selected sequence takes place step- by-step in each case in reaction response to a respective the first user input, and does so in an order which is predetermined by the presently selected sequence (Shastri Col. 6, lines 46-67 and note that display protocol sequences PS, there is a selection screen displayed on monitors 22A and 22B as shown In FIG. 4, from which it is possible to select by clicking with the mouse, a desired display protocol sequence PS. Further, at this time, it is beneficial that the display protocol sequence be selected corresponding to the examination, the series, the type of input modality 4 in which the images were formed (CR images, CT images, MRI images, etc.), the examining doctor or group of doctors to which the examining doctor belongs (reading doctor, clinical doctor, referring doctor, etc.) the place where the images are to be employed (an examination room, a doctor's residence, etc.), the purpose for which the images are to be used, (reading, reference, comparison, etc.), etc.(24) Then, by selecting the display protocol sequence from these types of display protocol sequences, controller 27 displays images on monitors 22A and 22B in the layout corresponding to the order of the display protocols in the selected display protocol sequence. That is, for PS1, display protocol P1 is first, followed by P2, which is then followed by P3. In other words, the display protocols are lined up and then displayed in a temporal manner.. When controller 27 receives transmission of image data S, the selected display protocol sequence is read out from memory 28, and at first, based on to the first display protocol P1, the layout is performed and the laid out images are displayed on monitors 22A and 22B.).
As of Claim 5: Shastri in view of Kohle in view of Zhao further teaches the visualization modes comprised by the selected sequence of different visualization modes is a subset of a larger number of overall visualization modes that are settable using the visualization system (1) (Shastri Col. 4, lines 40-67 and Col. 8, lines 1-3 note that there are multiple display protocol sequences, because the display protocol sequence can be switched, the examiner of images can display the plurality of images according to a desired display protocol sequence.).
As of Claim 6: Shastri in view of Kohle in view of Zhao further teaches determining or changing (160) the selected predefined sequence of different visualization modes based on at least one of: a second user input, stored user preferences, a specific system parameter, application parameter, or the received live image data (Shastri Col. 4, lines 40-67 and Col. 8, lines 1-3 note that there are multiple display protocol sequences, because the display protocol sequence can be switched, the examiner of images can display the plurality of images according to a desired display protocol sequence.).
As of Claim 7: Shastri in view of Kohle in view of Zhao further teaches the user or the computer device (3),at least one of changes or predefines one or more of the various sequences of visualization modes stored in the memory, to define a plurality of applicable sequences of visualization modes reasonable for a respective application (Shastri Col. 6, lines 46-67 and note that display protocol sequences PS, there is a selection screen displayed on monitors 22A and 22B as shown In FIG. 4).
As of Claim 8: Shastri in view of Kohle in view of Zhao further teaches the different visualization modes of one or more of the sequences of visualization modes stored in the memory are determined or changed by the computer device (3) (Shastri Col. 6, lines 46-67) by using at least one of artificial intelligence or machine learning based on at least one of:the visualization modes used last by the user,a medical application, or the received live image data (¶0038 of Kohle).
As of Claim 9: Shastri in view of Kohle in view of Zhao further teaches determining (160) the predefined sequence of various visualization modes based on a plurality of predefined various sequences of the different visualization modes which are stored in the memory (Shastri Col. 6, lines 46-67).
As of Claim 10: Shastri in view of Kohle in view of Zhao further teaches in reaction to a second user input, changing at least one of an imaging mode, a display layout, or individual parameters of at least one of the different visualization modes of the selected sequence of visualization modes in response to a second user input (Kohle ¶¶0057-0064).
As of Claim 11: Shastri in view of Kohle in view of Zhao further teaches the step of displaying(120) the received liveimage data (¶0034 of Zhao)in the first visualization mode (a) further comprises: displaying a preview representation (9) of the received liveimage data in a further visualization mode or in multiple further visualization modes of the predefined sequence (Shastri Col. 4, lines 40-67 and Col. 8, lines 1-3 note that there are multiple display protocol sequences, because the display protocol sequence can be switched, the examiner of images can display the plurality of images according to a desired display protocol sequence.).
As of Claim 12: Shastri in view of Kohle in view of Zhao further teaches all of the different visualization modes of the predefined selected sequence, or a visualization mode directly preceding or following the first visualization mode are displayed in the preview representation (9) (¶¶0024, 0045 of Zhao).
Claims 13, 14, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shastri (US 7259729 B2) in view of Zhao et al. (US 20230147826 A1; hereafter Zhao).
As of Claim 13: Shastri teaches a medical device visualization system (1) comprising a microscope, exoscope for displaying image data, the medical device visualization system (1) (Col. 4, lines 27-40 and note that FIG. 1, this medical image network and as shown in FIG. 2, workstation 2 is provided with a controller 27 and a memory 26. Note that in the current embodiment of the two monitors, 22A and 22B are used for displaying images.) comprising: a camera (2) for generating live image data; an imaging device (2) for receiving the image data;
a display device (5) for displaying the received image data (FIG. 3A, the display screen of monitors 22A and 22B are each divided into 9 sections, and are arranged and displayed in a predetermined temporal order. For example, image S2 is displayed first, followed by image S4. Here, images S2 and S4 each have 36 cross-sectional images, however, because only 18 images can be displayed at one time on monitors 22A and 22B, according to a switching operation of input means 23, the displaying of all the images is split into two times and carried out);
an input device (8) for receiving a user input ( input means 23); and a computer (3, 4) configured to carry out the method according to claim 1, the computer including said memory for storing the various sequences of visualization modes (display protocol sequence is read out from memory 28), with each visualization mode comprising said specific combination of the imaging mode and the associated display layout (Col. 6, lines 56-67 and note that note that for such display protocol sequences PS, there is a selection screen displayed on monitors 22A and 22B as shown In FIG. 4, from which it is possible to select by clicking with the mouse, a desired display protocol sequence PS. Further, at this time, it is beneficial that the display protocol sequence be selected corresponding to the examination, the series, the type of input modality 4 in which the images were formed (CR images, CT images, MRI images, etc.), and the selected sequence is stored in the memory and is selectable by the user ;wherein the visualization system (1) is configured, in response to the at least one user input, to make a change, in a predefined and currently selected sequence of visualization modes from the first(Col. 8, lines 23-37 and note that the display format is switched according to the generated display protocol sequence.).
Zhao is a similar or analogous system to the claimed invention as evidenced Zhao teaches improving surgeon's view and context awareness during laparoscopic and video assisted surgeries (Abstract) that would have prompted a predictable variation of Shastri by applying Zhao’s known principal of live image data; (¶0034 and note that screen layout 600, the left side may show CT, MRI, PET, ultrasound images and/or 3D model(s) 610 (e.g., in reconstructed 3D model), 612 (e.g., in CT axial view), 614 (e.g., in CT sagittal view), 616 (e.g., in CT coronal view) stored in storage 414, and the right side may show live laparoscopic images 620 from laparoscope 360), an endoscope (¶0044 of Zhao and note video-assisted thoracoscopic, endoscopic, percutaneous, colonoscopy surgeries, etc. Some or all of the disclosure relating to laparoscopes and laparoscopy may be similarly applied to other video-assisted surgical instruments and operations, such as endoscope and endoscopy, percutaneous scope(s) and surgery, colonoscope and colonoscopy, etc.).
In view of the motivations such as providing a user surgeon has sufficient knowledge to be able to label matching points or features between the left-side (e.g., static and/or live dynamic) content and the right-side (e.g., live dynamic) content shown in screen layout 600 thereby further improving surgeon's view during surgery, by utilizing surgeon's guidance sporadically to achieve high image registration accuracy of ¶0034 of Zhao that one of ordinary skill in the art would have implemented the claimed variation of the prior art system of Shastri.
Therefore, the claimed invention would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
As of Claim 14: Shastri in view of Zhao further teaches the camera is a video endoscope (¶0044 of Zhao and note video-assisted thoracoscopic, endoscopic, percutaneous, colonoscopy surgeries, etc. Some or all of the disclosure relating to laparoscopes and laparoscopy may be similarly applied to other video-assisted surgical instruments and operations, such as endoscope and endoscopy, percutaneous scope(s) and surgery, colonoscope and colonoscopy, etc.).
As of Claim 16: all the limitations are addressed in claim 13. Moreover, Shastri teaches non-transitory, computer-readable storage medium, comprising commands, which, upon execution by the computer device (3, 4) of the medical device visualization system (1) according to claim 13, prompt the computer device (3, 4) to carry out the method (¶0016).
NOTE: The Examiner strongly encourages setting up an interview to expedite prosecution of the case.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contacts
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEKONNEN D DAGNEW whose telephone number is (571)270-5092. The examiner can normally be reached on 8:00AM-5:00PM M-Th.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lin Ye can be reached on 571-272-7372. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEKONNEN D DAGNEW/Primary Examiner, Art Unit 2638