DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/02/2026 has been entered.
Response to Amendment and Arguments.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The newly modified claim limitation “detecting, within the speech of the user in the live audio stream, an indicator to a problem with the medical system, the indicator occurring at least during a time window; in response to detecting the indicator, selecting an audio segment within the live audio stream and at least one temporally corresponding data segment [[in]]from each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one temporally corresponding data segment comprising audio and data captured, respectively, during at least the time window; outputting the audio segment and the at least one temporally corresponding data segment; troubleshooting the problem using the audio segment and the at least one temporally corresponding data segment; and updating the medical system based on the troubleshooting to correct the problem.” necessitates the new ground of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4, 10-14, 16, 19, and 20 are rejected under 35 U.S.C. 103 as obvious over Itkowitz, (US 20200152190.) in view of Venkataraman, (US 20200372998).
Claim 1, 19, and 20
Regarding Claim 1, 19, and 20, Itkowitz teaches
A method comprising: obtaining during an operation involving a medical system, a live audio stream comprising speech by a user using the medical system for performing the operation, wherein the operation is performed upon a subject;
(“[0021] Referring to FIG. 1A of the drawings, a teleoperational medical system for use in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by the reference numeral 10. As will be described, the teleoperational medical systems of this disclosure are under the teleoperational control of a surgeon. In alternative embodiments, a teleoperational medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure. In still other alternative embodiments, a fully automated medical system, under the full control of a computer programmed to perform the procedure or sub-procedure, may be used to perform procedures or sub-procedures. As shown in FIG. 1A, the teleoperational medical system 10 generally includes a teleoperational assembly 12 mounted to or near an operating table O on which a patient P is positioned. The teleoperational assembly 12 may be referred to as a patient side cart. A medical instrument system 14 and an endoscopic imaging system 15 are operably coupled to the teleoperational assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view images of or representing the surgical site and to control the operation of the medical instrument system 14 and/or the endoscopic imaging system 15.”
“[0025] The control system 20 is in communication with or includes a speech recognition system 27. The speech recognition system 27 includes one or more microphones for receiving voice communications from personnel in the surgical environment, particularly the surgeon S. The speech recognition system may further include a one or more processors and one or more memory devices for processing the voice communications received by the microphone. Optionally, the processor 22 and memory 24 may process the voice communications received by the speech recognition system 27. The processors may include software and related hardware for receiving and interpreting voice communications from a surgeon and generating appropriate corresponding output signals. The microphones of the speech recognition system 27 may be located in in close proximity to the surgeon S or other surgical staff to reduce the amount of background noise provided to the processor. Optionally, one or more of the microphones may be mounted to a headset that is worn by the surgeon S or other surgical staff.
[0026] The speech recognition system 27 digitizes the oral voice communications received by the microphone, converting the voice communications into electronic form. The digitized words or sounds are analyzed and interpreted using natural language processing or other speech processing technologies. The analysis may include a comparison with a library of recognized words and sounds stored in in the memory of the speech recognition system or accessible to the speech recognition system over an internal network (e.g., a secured network of a medical facility or a teleoperational system provider) or an external network (e.g, the Internet).”
“[0037] At a process 102, a voice communication in the surgical environment is recognized by the control system (e.g., control system 20). More specifically, the speech recognition system 27 may detect voice communication from the surgeon S or another member of the surgical team. The detected voice communication is analyzed and interpreted by the speech recognition system 27 and/or the control system 20. U.S. Pat. No. 6,591,239 (filed Dec. 9, 1999) (disclosing “Voice Controlled Surgical Suite”) which is incorporated by reference herein in its entirety, discloses one such speech recognition system.”)
obtaining at least one data stream from equipment associated with the medical system; and
(“[0038] At a process 104, a variety of surgical environment state variables 200 may be monitored and assessed by the control system 20. The variables 200 provide information about the state of various systems, instruments, equipment, procedures, and people within the surgical environment. A speaker state variable 202 provides information about the speaker of the voice communication. The speaker may be anyone on the surgical team including the surgeon S and/or the surgical staff. The information about the speaker may include identification information, training history, credentials, procedure history, typical surgical team members, communication preferences, frequently used vernacular/jargon, anthropometric information, ergonomic preferences, equipment preferences, interface preferences, and the speaker's physical location in the surgical environment, including proximity to systems and instruments. The training history may include, for example, a cumulative record of the user's simulator experience and proctor- assisted procedure experience, including the types of procedures, the outcome of the procedures, and any issues occurring during the procedures. It may also include evaluations, certifications, and a cumulative log of hours in training. The training history may be updated after each training episode for a user. The credential information may include, for example, credentials or other rights to use the systems or to access specific procedures with those systems. Credentials may be issued by an issuing authority such as a trainer, a medical facility (e.g., a hospital, clinic, training center). The procedure history information may include, for example, a cumulative record of the procedures performed by the speaker including types of procedures, any user idiosyncrasies, procedure outcomes, and previously recognized voice communications. The procedure information may include a count of procedures performed, types of procedures performed, speed of procedures performed, and transition times for prior procedures. The procedure information may further include the software version and model of the system used for each prior procedure. The communication preferences may include a record of the languages in which the speaker is fluent and preferred languages for audio and/or textual communication. The communication preferences may also include the speaker's preferences regarding the medium for delivery of communication (e.g., visual, auditory, combined visual and auditory) and volume settings. The anthropometric information may include anatomic measurement information for the speaker including, for example, optometric measurements of vision and any needed corrective lenses, intraocular spacing, height, weight, handedness, and physical limitations including hearing or vision. The ergonomic preferences may include operator control and instrument settings that the speaker finds to be most comfortable or useful. The equipment preferences information may include the speaker's preferences regarding optional arrangements, functions, and settings of components of a teleoperational system (e.g., system 10, instrument system 14, user console 16). For example, the equipment preferences may include preferred hand positions and button/pedal function assignments for the control console 16. Preferences may include the speaker's preferred configuration of the assembly 12 relative to the patient. Preferences may include preferred instrument (e.g., instrument 14) settings such as ablation power levels, energy, force, torque, staple cartridge, and handedness for stapler. The preferences may include preferred port placements and arm configurations. The preferences may include preferred functionality such as preferred table angles, patient positioning presets, or microsurgery capability. The preferences may include preferred auxiliary equipment (e.g., equipment 26) including supplemental imaging systems (e.g., MM, x-ray, ultrasound); video input and output; insufflation settings (e.g., desire pressure, maximum flow rate), audio settings (e.g., which microphones activated, feedback suppression, which speakers activated, use of voice prompts). The user interface preferences may include the speaker's preferences regarding the graphical user interface, other sensory displays, or the endoscopic instrument settings. For example, preferences may relate to vision correction and autofocus. Preferences may also include the speaker's preferred display color, brightness, contrast, shadow, dynamic contrast, and use of near infrared imaging.”)
during the operation:
detecting, within the speech of the user in the live audio stream, an indicator to a problem with the medical system, the indicator occurring at least during a time window;
(“[0049] At a process 106, the voice communication is evaluated in the context of the surgical environment state variables 200. More specifically, one or more of the variables 200 are used, for example, to determine the meaning of the voice communication, answer a question posed by the voice communication, trouble-shoot a problem identified in the voice communication, execute a command made in the voice communication, resolve ambiguity raised in the voice communication, identify warnings associated with the voice communication, and/or provide auditory or textual instructions to another team member in the surgical environment. The meaning of the voice communication may be determined by reference to word recognition search space or library. Words in the word recognition search space may be promoted or prioritized for matching with the voice communication based on the assessed surgical environment state variables. Words in the word recognition search space are associated with output commands to the various components of the surgical system. The word recognition search space may be constrained by the surgical state variables so that system responses not associated with the variable constraints may be eliminated from consideration when determining a response.”
“[0060] In another example, if the voice communication, “the instrument won't move correctly” is received, variables associated with “instrument” and “won't move correctly” are evaluated in the context of multiple surgical state variables including the procedure 204, instruments 206, manipulator 208, visualization equipment 214, and master console 210. For example, an evaluation of the manipulator variables 208 may indicate that two manipulator arms have come into contact or that movement of one of the manipulator arms is impinged upon by another piece of equipment in the surgical environment or a range of motion limitation. Alternatively, an evaluation of the master console variables 210 may indicate that the operator control devices are attempting to move outside of a permitted range of motion or that another operator in a dual console system currently has control. Alternatively, an evaluation of the instrument variables 206 may indicate that the instrument is not properly engaged with the manipulator arm or that the attempted movement is outside the instrument range of motion. Alternatively, an evaluation of the procedure variables 204, manipulator variables 208, and/or the visualization equipment variables 214 may indicate that the endoscope manipulator arm is activated (e.g. clutched), thus deactivating the other instrument arms. If the evaluation of variables determines that the manipulator arms are contacting each other, the determined and implemented system response may be to clutch the manipulator arm and readjust, change to a different endoscopic viewing angle, adjust or create a new access port, swap instruments between manipulator arms, or swap manipulator arms between ports.
[0061] In another example, if the voice communication, “the ESU isn't working” is received, the acronym ESU may be recognized as referring to an electrosurgical unit (e.g. a type of auxiliary equipment). The variables 212, 210 associated with the electrosurgical unit may be evaluated to determine whether there is electrical power being provided to the unit; whether the power is set insufficiently high for the commanded procedure; whether the foot pedal control on the operator console is malfunctioning whether the effect level has not been set and therefore defaulted to zero; whether the energy cable between the instrument and ESU is connected; or whether the energy pedal is actuated while the operator's head is not detected at the console viewer.”)
troubleshooting the problem using the audio segment and the at least one temporally corresponding data segment;
(“[0049] At a process 106, the voice communication is evaluated in the context of the surgical environment state variables 200. More specifically, one or more of the variables 200 are used, for example, to determine the meaning of the voice communication, answer a question posed by the voice communication, trouble-shoot a problem identified in the voice communication, execute a command made in the voice communication, resolve ambiguity raised in the voice communication, identify warnings associated with the voice communication, and/or provide auditory or textual instructions to another team member in the surgical environment. The meaning of the voice communication may be determined by reference to word recognition search space or library. Words in the word recognition search space may be promoted or prioritized for matching with the voice communication based on the assessed surgical environment state variables. Words in the word recognition search space are associated with output commands to the various components of the surgical system. The word recognition search space may be constrained by the surgical state variables so that system responses not associated with the variable constraints may be eliminated from consideration when determining a response.”)
and updating the medical system based on the troubleshooting to correct the problem.
(“[0060] In another example, if the voice communication, “the instrument won't move correctly” is received, variables associated with “instrument” and “won't move correctly” are evaluated in the context of multiple surgical state variables including the procedure 204, instruments 206, manipulator 208, visualization equipment 214, and master console 210. For example, an evaluation of the manipulator variables 208 may indicate that two manipulator arms have come into contact or that movement of one of the manipulator arms is impinged upon by another piece of equipment in the surgical environment or a range of motion limitation. Alternatively, an evaluation of the master console variables 210 may indicate that the operator control devices are attempting to move outside of a permitted range of motion or that another operator in a dual console system currently has control. Alternatively, an evaluation of the instrument variables 206 may indicate that the instrument is not properly engaged with the manipulator arm or that the attempted movement is outside the instrument range of motion. Alternatively, an evaluation of the procedure variables 204, manipulator variables 208, and/or the visualization equipment variables 214 may indicate that the endoscope manipulator arm is activated (e.g. clutched), thus deactivating the other instrument arms. If the evaluation of variables determines that the manipulator arms are contacting each other, the determined and implemented system response may be to clutch the manipulator arm and readjust, change to a different endoscopic viewing angle, adjust or create a new access port, swap instruments between manipulator arms, or swap manipulator arms between ports.”
“[0070] At a process 310, the response to the voice communication is implemented based on the voice localization variable via the identified subsystem. Subsystem dependent responses may be limited to command sets associated with the subsystem. For example, subsystem dependent responses may include commands to authorize control of the subsystem or instruments attached to the subsystem; to change a setting (e.g., display brightness, audio volume); to mute/un-mute an intercom microphone; to show or hide status messages; to set a subsystem value (e.g., an illumination level or an insufflation pressure); retrieve a value (e.g., an insufflation pressure or a temperature); adjust a value (e.g., a display brightness or a speaker volume); initiate a configuration (e.g., a set-up configuration); set a display mode (e.g., tile display, fluorescent image, uni-ocular, stereoscopic); retrieve a status (e.g. a clock time, an elapsed time, a recording on/off status); to troubleshoot (e.g., “why can't I move this instrument”); to acknowledge system messages (e.g. confirm content of warning message); to perform an instrument exchange (e.g., eject a tool, release a grip, swap an arm); to perform another discrete action (e.g., flip the endoscope angle, level a view, take images, annotate images, start/stop recordings, adjust zoom, adjust camera position, adjust master/slave scaling; or to perform advanced controls (e.g., table motion, optimize manipulator position, initiate ergonomic settings). Implementing the response may also include disabling components associated with non-identified subsystems.”)
However, Itkowitz does not specifically disclose (the bolded) detecting, within the speech of the user in the live audio stream, an indicator to a problem with the medical system, the indicator occurring at least during a time window;
in response to detecting the indicator, selecting an audio segment within the live audio stream and at least one temporally corresponding data segment [[in]]from each data stream of the at least one data stream, temporally corresponding data segment comprising audio and data captured, respectively, during at least the time window;
outputting the audio segment and the at least one temporally corresponding data segment;
troubleshooting the problem using the audio segment and the at least one temporally corresponding data segment;
However, Venkataraman teach (the bolded )
detecting, within the speech of the user in the live audio stream, an indicator to a problem with the medical system, the indicator occurring at least during a time window;
(“[0053] In order to combine different modalities of data in a more meaningful manner, multiple modalities of data, e.g., video data, image data, audio data can be synchronized based on time (or “time-synced”). For example, if segmentation engine 106 for audio data can detect an event (e.g., locating a tumor in an anatomy) being discussed in an audio at, e.g., t.sub.0 timestamp, and segmentation engine 102 also detects the same event in an endoscope video at the same time mark, then a segment of the video and a segment of audio around t.sub.0 timestamp can be meaningfully combined. Moreover, if there are additional intra-op imaging taking place (e.g., CT scans taken with a mobile CT scanner or ultrasonic scans taken with an ultrasonic device) at t.sub.0 timestamp, these images can be meaningfully combined with the segment of the video and the segment of audio around t.sub.0 timestamp for making a collaborative decision, e.g., to determine the location of the tumor within an anatomy of the endoscope video with high confident.
[0054] In some embodiments, combining multiple modalities of data in a more meaningful manner can include synchronizing (or “syncing”) the multiple modalities of data based on an easily identifiable time event associated with a particular modality. For example, let's assume that an intra-op imaging process begins at t.sub.0 and takes 2 minutes to complete from t.sub.0 to t.sub.0+2-min, and there is also a corresponding audio that describes the surgical procedure, then it becomes more meaningful to combine the resulting intra-op images with a segment of audio from t.sub.0 to t.sub.0+2-min. Moreover, if there is also a corresponding endoscope video during the same time period of t.sub.0 to t.sub.0+2-min, it would be more beneficial to also combine the video clip from the same 2-minute segment with the corresponding imaging and audio data. Moreover, if there are also text data generated at the same time periods, those text data can also be meaningfully combined with the other modalities of data of the same time period.”)
in response to detecting the indicator, selecting an audio segment within the live audio stream and at least one temporally corresponding data segment [[in]]from each data stream of the at least one data stream, temporally corresponding data segment comprising audio and data captured, respectively, during at least the time window;
(“[0053] In order to combine different modalities of data in a more meaningful manner, multiple modalities of data, e.g., video data, image data, audio data can be synchronized based on time (or “time-synced”). For example, if segmentation engine 106 for audio data can detect an event (e.g., locating a tumor in an anatomy) being discussed in an audio at, e.g., t.sub.0 timestamp, and segmentation engine 102 also detects the same event in an endoscope video at the same time mark, then a segment of the video and a segment of audio around t.sub.0 timestamp can be meaningfully combined. Moreover, if there are additional intra-op imaging taking place (e.g., CT scans taken with a mobile CT scanner or ultrasonic scans taken with an ultrasonic device) at t.sub.0 timestamp, these images can be meaningfully combined with the segment of the video and the segment of audio around t.sub.0 timestamp for making a collaborative decision, e.g., to determine the location of the tumor within an anatomy of the endoscope video with high confident.
[0054] In some embodiments, combining multiple modalities of data in a more meaningful manner can include synchronizing (or “syncing”) the multiple modalities of data based on an easily identifiable time event associated with a particular modality. For example, let's assume that an intra-op imaging process begins at t.sub.0 and takes 2 minutes to complete from t.sub.0 to t.sub.0+2-min, and there is also a corresponding audio that describes the surgical procedure, then it becomes more meaningful to combine the resulting intra-op images with a segment of audio from t.sub.0 to t.sub.0+2-min. Moreover, if there is also a corresponding endoscope video during the same time period of t.sub.0 to t.sub.0+2-min, it would be more beneficial to also combine the video clip from the same 2-minute segment with the corresponding imaging and audio data. Moreover, if there are also text data generated at the same time periods, those text data can also be meaningfully combined with the other modalities of data of the same time period.”)
outputting the audio segment and the at least one temporally corresponding data segment;
(“[0061] Process 300 may begin by receiving a set of intra-op images generated from an intra-op imaging process during a surgical procedure of a region of interest within a patient's body (step 302). For example, the intra-op imaging process can include a CT scan; an X-ray scan; an MRI scan, an ultrasonic scan, among others. Process 300 next determines a beginning timestamp t.sub.1 and an ending timestamp t.sub.2 for the intra-op imaging process (step 304). Next, process 300 receives a recorded OR audio containing surgeon's narration of the surgical procedure and an endoscope video of the actual surgical procedure (step 306). Process 300 then selects an audio segment between t.sub.1 and t.sub.2 from the OR audio and a video segment between t.sub.1 and t.sub.2 from the endoscope video (step 308). Next, process 300 provides the set of intra-op images, the audio segment, and the video segment to a feature extraction engine to extract and combine multiple sets of text features into a combined set of features in an N-dimensional feature space (step 310). Note that the feature extraction engine in step 310 can be implemented with comprehensive feature generation module 142 described in conjunction with FIG. 1.”)
troubleshooting the problem using the audio segment and the at least one temporally corresponding data segment;
(“[0053] In order to combine different modalities of data in a more meaningful manner, multiple modalities of data, e.g., video data, image data, audio data can be synchronized based on time (or “time-synced”). For example, if segmentation engine 106 for audio data can detect an event (e.g., locating a tumor in an anatomy) being discussed in an audio at, e.g., t.sub.0 timestamp, and segmentation engine 102 also detects the same event in an endoscope video at the same time mark, then a segment of the video and a segment of audio around t.sub.0 timestamp can be meaningfully combined. Moreover, if there are additional intra-op imaging taking place (e.g., CT scans taken with a mobile CT scanner or ultrasonic scans taken with an ultrasonic device) at t.sub.0 timestamp, these images can be meaningfully combined with the segment of the video and the segment of audio around t.sub.0 timestamp for making a collaborative decision, e.g., to determine the location of the tumor within an anatomy of the endoscope video with high confident.
[0054] In some embodiments, combining multiple modalities of data in a more meaningful manner can include synchronizing (or “syncing”) the multiple modalities of data based on an easily identifiable time event associated with a particular modality. For example, let's assume that an intra-op imaging process begins at t.sub.0 and takes 2 minutes to complete from t.sub.0 to t.sub.0+2-min, and there is also a corresponding audio that describes the surgical procedure, then it becomes more meaningful to combine the resulting intra-op images with a segment of audio from t.sub.0 to t.sub.0+2-min. Moreover, if there is also a corresponding endoscope video during the same time period of t.sub.0 to t.sub.0+2-min, it would be more beneficial to also combine the video clip from the same 2-minute segment with the corresponding imaging and audio data. Moreover, if there are also text data generated at the same time periods, those text data can also be meaningfully combined with the other modalities of data of the same time period.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Itkowitz to incorporate the teachings of Venkataraman to provide a time window Doing so would make the data more meaningful, as recognized by Venkataraman. (Paragraph 53-54).
Regarding claim 19 Itkowitz Further teaches, A computerized apparatus having a processor coupled with a memory unit, the processor being adapted to perform the steps of:
(“[0024] The teleoperational medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22, and typically a plurality of processors, for effecting control between the medical instrument system 14, the operator input system 16, and other auxiliary systems 26 which may include, for example, imaging systems, audio systems (including an intercom system), fluid delivery systems, display systems, mobile vision carts, illumination systems, steering control systems, irrigation systems, and/or suction systems. The control system 20 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. While control system 20 is shown as a single block in the simplified schematic of FIG. 1A, the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the teleoperational assembly 12, another portion of the processing being performed at the operator input system 16, and the like. Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the teleoperational systems described herein. In one embodiment, control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.”)
Regarding claim 20 YANG Further teaches A computer program product comprising a non-transitory computer readable medium retaining program instruction, which instructions when read by a processor, cause the processor to perform:
(“[0024] The teleoperational medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22, and typically a plurality of processors, for effecting control between the medical instrument system 14, the operator input system 16, and other auxiliary systems 26 which may include, for example, imaging systems, audio systems (including an intercom system), fluid delivery systems, display systems, mobile vision carts, illumination systems, steering control systems, irrigation systems, and/or suction systems. The control system 20 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. While control system 20 is shown as a single block in the simplified schematic of FIG. 1A, the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the teleoperational assembly 12, another portion of the processing being performed at the operator input system 16, and the like. Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the teleoperational systems described herein. In one embodiment, control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.”)
Claim 2
Regarding Claim 2, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 2. (Previously Presented) The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the live audio stream at least one word selected from a predetermined list of words.
(“[0049] At a process 106, the voice communication is evaluated in the context of the surgical environment state variables 200. More specifically, one or more of the variables 200 are used, for example, to determine the meaning of the voice communication, answer a question posed by the voice communication, trouble-shoot a problem identified in the voice communication, execute a command made in the voice communication, resolve ambiguity raised in the voice communication, identify warnings associated with the voice communication, and/or provide auditory or textual instructions to another team member in the surgical environment. The meaning of the voice communication may be determined by reference to word recognition search space or library. Words in the word recognition search space may be promoted or prioritized for matching with the voice communication based on the assessed surgical environment state variables. Words in the word recognition search space are associated with output commands to the various components of the surgical system. The word recognition search space may be constrained by the surgical state variables so that system responses not associated with the variable constraints may be eliminated from consideration when determining a response.
[0050] Evaluating the voice communication in the context of the surgical environment state variables may include limiting a word recognition search space based upon the variables. For example, if the assessment of the instrument surgical state variable indicates that the instruments in the surgical space are graspers and cautery shears only, the term “sealer” may be eliminated from word recognition search space to avoid confusion between the terms “shears” and “sealer.” As another example, if the assessment of the instrument surgical state variable indicates that a monopolar curved scissors is in use, alternative names and known jargon such as “MCS,” “scissors,” “shears,” “hot shears,” and “cautery shears” are prioritized as potential matches with the recognized voice communication.”)
Claim 4
Regarding Claim 4, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Venkataraman teaches 4. (Currently Amended) The method of Claim 1, wherein the audio segment and the at least one temporally corresponding data segment comprise, respectively, audio and data captured during a second time window preceding the time window.
(“[0053] In order to combine different modalities of data in a more meaningful manner, multiple modalities of data, e.g., video data, image data, audio data can be synchronized based on time (or “time-synced”). For example, if segmentation engine 106 for audio data can detect an event (e.g., locating a tumor in an anatomy) being discussed in an audio at, e.g., t.sub.0 timestamp, and segmentation engine 102 also detects the same event in an endoscope video at the same time mark, then a segment of the video and a segment of audio around t.sub.0 timestamp can be meaningfully combined. Moreover, if there are additional intra-op imaging taking place (e.g., CT scans taken with a mobile CT scanner or ultrasonic scans taken with an ultrasonic device) at t.sub.0 timestamp, these images can be meaningfully combined with the segment of the video and the segment of audio around t.sub.0 timestamp for making a collaborative decision, e.g., to determine the location of the tumor within an anatomy of the endoscope video with high confident.”)
See claim one for rationale.
Claim 10
Regarding Claim 10, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 10. (Original) The method of Claim 1, wherein the at least one data stream comprises data related to actions performed by the user.
(“[0041] The instrument state variable 206 includes information about the instrument (e.g., instrument 14) or instruments for past, current, or future use in the surgical procedure. The information may include instrument identification information, configurations, operational settings, and common failure modes. The instrument state variable 206 may include information about alternative names used to identify instruments, the instrument range of motion, and kinematic information such as the current location of the instrument tips. The instrument state variable 206 may include information about what an instrument is currently doing and whether a command associated with a voice communication is feasible or would cause damage to the patient or another portion of the surgical system.
[0042] The manipulator state variable 208 includes information about the teleoperational manipulator (e.g., manipulator 12) including, for example, the configuration of each arm, the movement range of each arm, the instrument attached to each manipulator arm, and common failure modes for the manipulator. The variable 208 may also include information about the range of motion of a manipulator arm and whether motion is obstructed by another object in the surgical environment.
[0043] The operator console state variable 210 includes information about the operator input system (e.g., system 16) including, for example, information about the functional assignment of the control devices 36, 37, the degrees of freedom of movement associated with each control device, the images visible through the eye displays 32, 34, the range of movement for each control device, and common failure modes for the control devices or other aspects of the operator input system. The variable 210 may further include information about the volume or mute status of any speakers in the operator input system, whether a dual operator in system is in use and which station is currently in control,”
“[0060] In another example, if the voice communication, “the instrument won't move correctly” is received, variables associated with “instrument” and “won't move correctly” are evaluated in the context of multiple surgical state variables including the procedure 204, instruments 206, manipulator 208, visualization equipment 214, and master console 210. For example, an evaluation of the manipulator variables 208 may indicate that two manipulator arms have come into contact or that movement of one of the manipulator arms is impinged upon by another piece of equipment in the surgical environment or a range of motion limitation. Alternatively, an evaluation of the master console variables 210 may indicate that the operator control devices are attempting to move outside of a permitted range of motion or that another operator in a dual console system currently has control. Alternatively, an evaluation of the instrument variables 206 may indicate that the instrument is not properly engaged with the manipulator arm or that the attempted movement is outside the instrument range of motion. Alternatively, an evaluation of the procedure variables 204, manipulator variables 208, and/or the visualization equipment variables 214 may indicate that the endoscope manipulator arm is activated (e.g. clutched), thus deactivating the other instrument arms. If the evaluation of variables determines that the manipulator arms are contacting each other, the determined and implemented system response may be to clutch the manipulator arm and readjust, change to a different endoscopic viewing angle, adjust or create a new access port, swap instruments between manipulator arms, or swap manipulator arms between ports.”)
Claim 11
Regarding Claim 11, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 11. (Original) The method of Claim 10, wherein the actions performed by the user comprise user interface activation actions.
(“[0072] At a process 402, a speech recognition system enablement signal is received. The speech recognition system enablement signal may be, for example, a spoken trigger word or engagement of a physical trigger. For example the existing master clutch finger switch may be engaged to enable the speech recognition system. The master clutch finger switch may be located on one of the input control devices 36. The typical function of the master clutch finger switch is to interrupt the control loop linking the master control movement with the slave movement. This interruption allows the control devices to be repositioned. In this context, activation of the master clutch finger switch may have a secondary effect, namely that of enabling the speech recognition system. Optionally, following the activation of the master clutch finger switch, an uncharacteristic action or lack of action may further be required to enable the speech recognition system. For example, characteristically, activation of the master clutch finger switch is followed by repositioning of the control devices 36. If a predetermined period of time elapses without movement of the control devices 36, the activation of the master clutch finger switch may be recognized as a signal to enable the speech recognition system. Alternatively, the speech recognition system may be activated upon actuation of the master clutch but suspend activation (i.e., ignore speech) when the control system observes displacement of the control device beyond a threshold displacement value. Ignoring speech communication during active master clutch motion prevents acting upon erroneous or unintentional speech while also avoiding generating error feedback if partial or unrecognized speech detected. In other embodiments, an audible tone may be provided to alert the user that the speech recognition system is enabled and listening. In some embodiments, the master clutch finger switch is held is an activated state while the speech recognition system is enabled and listening.
[0073] At a process 404, a voice communication in the surgical environment is recognized by the control system (e.g., control system 20). More specifically, the speech recognition system 27 may detect voice communication from the surgeon S or another member of the surgical team. The detected voice communication is analyzed and interpreted by the speech recognition system 27 and/or the control system 20.
[0074] At a process 406, the response to the voice communication is implemented. In various embodiments, implementing the response may include suppressing other components of the system 10. For example, when the master clutch finger switch is activated at the surgeon console 16, the surgical suite intercom system may be suppressed so that spoken voice communication intended to provide system voice control is not broadcast to personnel in the surgical environment. This suppression avoids confusing the surgical personnel and reduces the risk that the surgical personnel will hear the commands and attempt to take corresponding action. In another alternative, a dedicated switch may activate a menu system that enables the detection of voice communication.”)
Claim 12
Regarding Claim 12, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 12. (Original) The method of Claim 1, wherein the at least one data stream comprises data related to actions taken by the system.
(“[0054] At a process 108, the system response to the recognized voice communication is determined based on one or more of the surgical environment state variables. The appropriate system response may be determined to be, for example, a command to control the motion of an instrument, to control motion of a manipulator arm, to control operation of auxiliary equipment, to make an adjustment to the endoscope, to send a textual or voice communication to a surgical staff member or another user, to update a patient record, to provide one or more follow-up inquiries to the speaker (e.g., via voice or text communication) to resolve ambiguity in or clarify/confirm the original voice communication. Determining the system response may include developing and presenting choices of system response to the speaker in order based on a confidence factor associated with a plurality of candidate responses.
[0055] At a process 110, the determined system response is implemented with one or more commands to one or more subsystems of the surgical system. For example, the determined system response may be implemented via a command 112 to control an instrument, a command 114 to control a textual or auditory communication to a user (including a user not present in the surgical area), a command 116 to control a manipulator arm, a command 118 to control a user control device, a command 120 to control the operation of auxiliary equipment, and/or a command 122 to control visualization equipment including the endoscope.”
“[0070] At a process 310, the response to the voice communication is implemented based on the voice localization variable via the identified subsystem. Subsystem dependent responses may be limited to command sets associated with the subsystem. For example, subsystem dependent responses may include commands to authorize control of the subsystem or instruments attached to the subsystem; to change a setting (e.g., display brightness, audio volume); to mute/un-mute an intercom microphone; to show or hide status messages; to set a subsystem value (e.g., an illumination level or an insufflation pressure); retrieve a value (e.g., an insufflation pressure or a temperature); adjust a value (e.g., a display brightness or a speaker volume); initiate a configuration (e.g., a set-up configuration); set a display mode (e.g., tile display, fluorescent image, uni-ocular, stereoscopic); retrieve a status (e.g. a clock time, an elapsed time, a recording on/off status); to troubleshoot (e.g., “why can't I move this instrument”); to acknowledge system messages (e.g. confirm content of warning message); to perform an instrument exchange (e.g., eject a tool, release a grip, swap an arm); to perform another discrete action (e.g., flip the endoscope angle, level a view, take images, annotate images, start/stop recordings, adjust zoom, adjust camera position, adjust master/slave scaling; or to perform advanced controls (e.g., table motion, optimize manipulator position, initiate ergonomic settings). Implementing the response may also include disabling components associated with non-identified subsystems.”)
Claim 13
Regarding Claim 13, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 13. (Original) The method of Claim 12, wherein the actions are taken by the system in response to user actions.
(“[0054] At a process 108, the system response to the recognized voice communication is determined based on one or more of the surgical environment state variables. The appropriate system response may be determined to be, for example, a command to control the motion of an instrument, to control motion of a manipulator arm, to control operation of auxiliary equipment, to make an adjustment to the endoscope, to send a textual or voice communication to a surgical staff member or another user, to update a patient record, to provide one or more follow-up inquiries to the speaker (e.g., via voice or text communication) to resolve ambiguity in or clarify/confirm the original voice communication. Determining the system response may include developing and presenting choices of system response to the speaker in order based on a confidence factor associated with a plurality of candidate responses.
[0055] At a process 110, the determined system response is implemented with one or more commands to one or more subsystems of the surgical system. For example, the determined system response may be implemented via a command 112 to control an instrument, a command 114 to control a textual or auditory communication to a user (including a user not present in the surgical area), a command 116 to control a manipulator arm, a command 118 to control a user control device, a command 120 to control the operation of auxiliary equipment, and/or a command 122 to control visualization equipment including the endoscope.”
“[0075] FIG. 5 illustrates a schematic view of a teleoperational medical system 500 comprising multiple discrete subsystems 502, 504, 506, 508, 510 responsive to and in communication with a control system 512 (e.g., system 20) that includes or is in communication with a speech recognition system 514 (e.g., system 27). The subsystems 502, 504, 506 may be, for example, teleoperational assemblies substantially similar to a teleoperational assembly 12 and may include one or a plurality of teleoperational arms. The subsystems 508, 510 may be, for example, operator input systems substantially similar to input system 16. Additional or alternative subsystems may include a display system, a mobile computing device, or an auxiliary system (e.g., system 26). An operator of the system 500 may issue voice commands recognized by the speech recognition system 514 (as previously described for system 27). Based on the recognized voice commands, the subsystems 502-510 may be operated discretely. For example a recognized voice command of “Eject arm 1” may cause the control system 512 to initiate the ejection of the instrument from a teleoperational manipulator at subsystem 502, the subsystem identified as including “arm 1”). Based on the recognized voice commands, the one or all of the subsystems 502-510 may be operated in combination. For example a recognized voice command of “Optimize positioning” may cause the control system 512 to simultaneously or sequentially move the arms of teleoperational manipulators of subsystems 502, 504, 506 to positions and orientations determined to be optimal for the present teleoperational procedure.
[0076] The recognized voice communication may be assessed in the context of trigger words alone or together with surgical environment state variables. For example, trigger words “arm 1,” “arm 2,” and “arm 3” may be associated with subsystems 502, 504, 506, respectively so that voice commands that include those words will be evaluated in the context of the identified subsystem and responses will be implemented in the context of the identified subsystem. Alternatively, the recognized voice communication may be assessed fully in the context of surgical environment state variables, including those associated with subsystems 502-510. example, system monitoring of the surgical state variables allows the recognized voice communication to be assessed in the context of the surgical state variables. If the operator commands, “Swap needle driver,” the monitored system state variables will indicate which subsystem 502-510 is operating a needle driver so that the response to the command is implemented on the that subsystem. If the operator commands, “optimize arm positions,” the monitored system state variables, providing position and orientation information about each other subsystems, will generate a response that may command a plurality of the subsystems to adjust.”)
Claim 14
Regarding Claim 14, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Itkowitz teaches 14. (Original) The method of Claim 13, wherein the at least one data stream comprises data related to measurements taken by the system.
(“[0034] FIG. 1C is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a left eye display 32 and a right eye display 34 for presenting the surgeon S with a coordinated stereo view of the surgical environment that enables depth perception. The console 16 further includes one or more input control devices 36, which in turn cause the teleoperational assembly 12 to manipulate one or more instruments or the endoscopic imaging system. The input control devices 36 can provide the same degrees of freedom as their associated instruments 14 to provide the surgeon S with telepresence, or the perception that the input control devices 36 are integral with the instruments 14 so that the surgeon has a strong sense of directly controlling the instruments 14. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the instruments 14 back to the surgeon's hands through the input control devices 36. Input control devices 37 are foot pedals that receive input from a user's foot.”
“[0041] The instrument state variable 206 includes information about the instrument (e.g., instrument 14) or instruments for past, current, or future use in the surgical procedure. The information may include instrument identification information, configurations, operational settings, and common failure modes. The instrument state variable 206 may include information about alternative names used to identify instruments, the instrument range of motion, and kinematic information such as the current location of the instrument tips. The instrument state variable 206 may include information about what an instrument is currently doing and whether a command associated with a voice communication is feasible or would cause damage to the patient or another portion of the surgical system.
[0042] The manipulator state variable 208 includes information about the teleoperational manipulator (e.g., manipulator 12) including, for example, the configuration of each arm, the movement range of each arm, the instrument attached to each manipulator arm, and common failure modes for the manipulator. The variable 208 may also include information about the range of motion of a manipulator arm and whether motion is obstructed by another object in the surgical environment.
[0043] The operator console state variable 210 includes information about the operator input system (e.g., system 16) including, for example, information about the functional assignment of the control devices 36, 37, the degrees of freedom of movement associated with each control device, the images visible through the eye displays 32, 34, the range of movement for each control device, and common failure modes for the control devices or other aspects of the operator input system. The variable 210 may further include information about the volume or mute status of any speakers in the operator input system, whether a dual operator in system is in use and which station is currently in control,
[0044] The auxiliary equipment state variable 212 includes information about the auxiliary equipment (e.g., systems 26) which may include configuration, setting, power, and failure mode information about imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and/or suction systems in use in the surgical environment.
[0045] The visualization equipment state variable 214 includes information about the endoscopic imaging system (e.g., system 15) and any associated display systems. The information may include, for example, pose information about the distal end of the endoscope in the patient anatomy, illumination settings, image processor settings, heat discharge information, power status, optical configuration, and common failure modes.”)
Claim 16
Regarding Claim 16, Itkowitz in view of Venkataraman teaches the limitations of claim 1. Further Venkataraman teaches 16. (Previously Presented) The method of Claim 1, wherein the at least one data stream comprises at least one type of data selected from the group consisting of: raw Electrocardiogram (ECG) digital sample recordings collected from ECG electrodes; ECG processed data; Intra- cardiac catheter navigation recordings; Intra-cardiac catheter shaft visualization; catheter localization and visualization algorithmic computation results; localization based on electromagnetic signals; localization based on currents; therapeutic radio frequency ablation or pulse field ablation applications including time duration or repetition information recordings; recording of a measurement of force applied; user action log, including recording of actions performed by the user with a user interface module; keyboard, mouse, or touch screen actions; and ultrasound frames or anatomical markers.
(“[0043] As can be seen in FIG. 1, surgical-data analytics system 100 receives video data 112, image data 114, and audio data 116 as inputs. Video data 112 can include endoscopy procedure videos capturing during the actual surgical procedure, and surgical procedure videos captured by one or more OR cameras installed near (e.g., directly above) the surgical site. Moreover, video data 112 can include both real-time video feed (if the data analytics is performed in real-time during the surgical procedure) and offline videos (if the data analytics if performed after the surgical procedure). Image data can include X-ray images, CT images, MRI images, ultrasonic images, among others, and each type of image data can be generated from a pre-op preparation/planning stage of the surgical procedure, from intra-op real-time imaging, or from a post-op time period after the surgical procedure. Moreover, image data 114 can include both real-time images (if the data analytics is performed in real-time during the surgical procedure) and offline images (if the data analytics if performed after the surgical procedure). Audio data 116 for a surgery procedure can include recorded audio files of a surgeon narrating/discussing the surgical procedure, such as regarding an anatomy, an event, a complication, among others.
[0044] Surgical-data analytics system 100 also includes a set of machine-learning-based segmentation engines 102-106 for converting each of video data 112, image data 114, and audio data 116 into a corresponding set of text features. More specifically, each of segmentation engines 102-106 is configured to segment a corresponding modality of surgical data 112-116 and extract a corresponding set of text features 122-126 from the corresponding surgical data. For example, segmentation engine 102 can be configured to convert a video segment in video data 112 into a first array of alphanumeric values representing the anatomy shown in the video segment. Meanwhile, segmentation engine 104 can be configured to convert a set of radiographic images in image data 114 into a second array of alphanumeric values representing one or more objects detected in the set of radiograph images. Moreover, segmentation engine 106 can be configured to convert an audio clip in audio data 116 into a third array of alphanumeric values representing an anatomy, a concern, a complication, or a landmark event extracted from the surgeon's narration/discussion.”
“[0053] In order to combine different modalities of data in a more meaningful manner, multiple modalities of data, e.g., video data, image data, audio data can be synchronized based on time (or “time-synced”). For example, if segmentation engine 106 for audio data can detect an event (e.g., locating a tumor in an anatomy) being discussed in an audio at, e.g., t.sub.0 timestamp, and segmentation engine 102 also detects the same event in an endoscope video at the same time mark, then a segment of the video and a segment of audio around t.sub.0 timestamp can be meaningfully combined. Moreover, if there are additional intra-op imaging taking place (e.g., CT scans taken with a mobile CT scanner or ultrasonic scans taken with an ultrasonic device) at t.sub.0 timestamp, these images can be meaningfully combined with the segment of the video and the segment of audio around t.sub.0 timestamp for making a collaborative decision, e.g., to determine the location of the tumor within an anatomy of the endoscope video with high confident.
[0054] In some embodiments, combining multiple modalities of data in a more meaningful manner can include synchronizing (or “syncing”) the multiple modalities of data based on an easily identifiable time event associated with a particular modality. For example, let's assume that an intra-op imaging process begins at t.sub.0 and takes 2 minutes to complete from t.sub.0 to t.sub.0+2-min, and there is also a corresponding audio that describes the surgical procedure, then it becomes more meaningful to combine the resulting intra-op images with a segment of audio from t.sub.0 to t.sub.0+2-min. Moreover, if there is also a corresponding endoscope video during the same time period of t.sub.0 to t.sub.0+2-min, it would be more beneficial to also combine the video clip from the same 2-minute segment with the corresponding imaging and audio data. Moreover, if there are also text data generated at the same time periods, those text data can also be meaningfully combined with the other modalities of data of the same time period.”)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Itkowitz to incorporate the teachings of Venkataraman to provide a “detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;” Doing so would determine the location of the tumor within an anatomy of the endoscope video with high confident, as recognized by Rosenblatt. (Paragraph 53).
Claims 5-9 are rejected under 35 U.S.C. 103 as obvious over Itkowitz in view of Venkataraman in further view of US 20130093829 A1, (Rosenblatt; Peter L)
Claim 5
Regarding Claim 5, Itkowitz in view of Venkataraman do not explicitly teach 5. (Original) The method of Claim 4, wherein the second time window starts at a predetermined event.
However, Rosenblatt further teaches
. The method of Claim 4, wherein the second time window starts at a predetermined event.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 110 "The time stamp data may include logged information, such as, but not limited to a start time stamp for the medical procedure, a start time stamp for each step in the medical procedure, an end time stamp for each step in the medical procedure, an end time stamp for the medical procedure, a start time stamp for an interaction with a specialist during the medical procedure, an end time stamp for an interaction with a specialist during the medical, and other logged information with a time stamp.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Itkowitz in view of Venkataraman to incorporate the teachings of Rosenblatt to provide a “5. (Original) The method of Claim 4, wherein the second time window starts at a predetermined event.” Doing so would ensure effective completion of procedures, as recognized by Rosenblatt. (Paragraph 5).
Claim 6
Regarding Claim 6, Rosenblatt further teaches
The method of Claim 4, wherein the second time window starts at a beginning time of the operation.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 110 "The time stamp data may include logged information, such as, but not limited to a start time stamp for the medical procedure, a start time stamp for each step in the medical procedure, an end time stamp for each step in the medical procedure, an end time stamp for the medical procedure, a start time stamp for an interaction with a specialist during the medical procedure, an end time stamp for an interaction with a specialist during the medical, and other logged information with a time stamp.")
See claim 5 for rationale.
Claim 7
Regarding Claim 7, Rosenblatt further teach
The method of Claim 1, wherein the audio segment and the at least one temporally corresponding data segment comprise, respectively, audio and data captured during a third time window consecutive to the time window.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 158 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems for scheduling a plurality of resources that may comprise logging times for steps taken in a step-by-step workflow associated with a procedure and a resource, projecting the completion time for the workflow, and generating a schedule for the resource based on the projected completion time for the workflow. By way of example, a practitioner may utilize an instruction system wherein a tablet device may be used to log time stamp data and/or other data associated with a procedure, such as performing an EKG. As the practitioner completes each step, the completion time may be logged and a projected time for the EKG machine to be available may be computed such that schedule may be generated based on said completion time.")
See claim 5 for rationale.
Claim 8
Regarding Claim 8, Rosenblatt further teach
The method of Claim 7, wherein the third time window ends at a predetermined event or at an end time of the operation.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 158 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems for scheduling a plurality of resources that may comprise logging times for steps taken in a step-by-step workflow associated with a procedure and a resource, projecting the completion time for the workflow, and generating a schedule for the resource based on the projected completion time for the workflow. By way of example, a practitioner may utilize an instruction system wherein a tablet device may be used to log time stamp data and/or other data associated with a procedure, such as performing an EKG. As the practitioner completes each step, the completion time may be logged and a projected time for the EKG machine to be available may be computed such that schedule may be generated based on said completion time.")
See claim 5 for rationale.
Claim 9
Regarding Claim 9, Rosenblatt further teaches
The method of Claim 1, further comprising concealing identifying details of the subject or of the user within the audio segment and the at least one temporally corresponding data segment.
(“[0138] Methods and systems of procedure instruction as described herein may also include or facilitate patient or practitioner host privacy through the use of video and/or audio obfuscation, such as facial obfuscation of a patient. The facial obfuscation may be enabled in various aspects of the procedure instruction system such as the video input portion, the video processing portion, the communication portion, and the like. In an example a practitioner host may be performing a procedure on a patient in an operating room and a specialist may be scheduled to interact through the audio and video capabilities described herein with the practitioner host during the procedure. To provide the optimal support for the practitioner host, the specialist may receive live video from the operating room including the use of the medical device, the procedure, the practitioner host, and the patient. Images that include the patient's face, for example, may include automatic obfuscation of the patient's face to prevent any possible recognition of the patient that may be detectable from patient's face. Obfuscation may also apply to other individuals in the procedure environment, such as medical assistance and the like. Obfuscation may be desirable for compliance with laws, such as the Health Insurance Portability and Accountability Act ("HIPAA").”
“[0155] Methods and systems may guide a practitioner to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems that may facilitate privacy through the use of video and/or audio obfuscation in various aspects of the procedure instruction system. By way of example, as a practitioner uses the procedure instruction system in a procedure environment to monitor technique and the timing of steps as completed by the practitioner, the practitioner may communicate with a specialist during the procedure who receives live video from the operating room, and the face or other identifying information of the patient may be blurred or otherwise blocked out to prevent the specialist from viewing such identifying information of the patient as the specialist assists the practitioner in using a particular medical device.”)
See claim 5 for rationale.
Claims 3 are rejected under 35 U.S.C. 103 as obvious over Itkowitz in view of Venkataraman in further view of US 20230185524 A1, (KOKER; Ekin).
Claim 3
Regarding Claim 3 Itkowitz in view of Venkataraman teach do not explicitly teach all of 3. The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.
However, KOKER teach
The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.
(Paragraph 77 "Likewise, the operation 212 analyzes the audio of the voice of the health professional to determine actionable information about the medical professional. For example, voice analysis of the medical professional’s voice may be applied to detect signs of tiredness, anger, depression, or so forth, possibly indicative of burnout, disinterest and boredom which may indicate a need for greater challenges, diversity of cases, opportunities for growth and development, signs of continuous or constant stress may indicate a need for additional training, and so forth. In another example, speech recognition (i.e., NLP) can be used to identify issues such as detecting abusive, inappropriate language, abrasive behaviors towards patients or fellow staff members, recognize voice commands to trigger calls to other patients (i.e., a radiology, remote expert, or so forth), initiate emergency protocols when help is needed (call 911, alert security, and so forth), recognize workflow stages based on words or phrases detected (e.g., “last scan,” “five minutes left in the examination, etc.), create timelines based on recognition of workflow activities, and so forth. As one illustrative example, voice analytics may be performed on speech of the medical professional in the audio to detect stress or exhaustion of the medical professional. In another example, the voice analytics may detect slurred speech of the medical professional indicative of impairment by alcohol or drugs. Again, these are merely nonlimiting illustrative examples.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Itkowitz in view of Venkataraman to incorporate the teachings of KOKER to provide a “3. The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.” Doing so would Detect abusive doctors, as well as impaired doctors , as recognized by KOKER. (Paragraph 77).
Claims 15 and 17 are rejected under 35 U.S.C. 103 as obvious over Itkowitz in view of Venkataraman in further view of US 20230225800 A1 , (Grund; Alessa Tabea).
Claim 15 and 17
Regarding Claim 15 and 17 Itkowitz in view of Venkataraman teach do not explicitly teach all of the method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.
However, Grund teach
The method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.
(paragraph 20 "FIG. 1A shows one embodiment of a combined cardiac electrophysiological mapping (EP), pacing and ablation system 100;
Some Embodiments of Electrophysiological Mapping and Pacing and Ablation Systems, Computer Systems, Intracardiac Imaging and Navigation Systems, and Body Surface Imaging and Navigation Systems")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Itkowitz in view of Venkataraman to incorporate the teachings of Grund to provide a “The method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.” Doing so would Result in higher rates of success in treating cardiac rhythm disorders, as recognized by Grund . (Paragraph 3).
Claims 18 are rejected under 35 U.S.C. 103 as obvious over Itkowitz in view of Venkataraman in further view of US 20220370246 A1 , (Akahoshi; Takayuki ).
Claim 18
Regarding Claim 18 Itkowitz in view of Venkataraman teach do not explicitly teach all of 18. The method of Claim 1, wherein the operation is a phacoemulsification operation.
However, Akahoshi teach
The method of Claim 1, wherein the operation is a phacoemulsification operation.
(Paragraph 83 "FIGS. 23-28 illustrate additional embodiments of phacoemulsification needles according to the present invention containing internal elements or projections within the tip portion of the needle for improving the efficiency of the needle and minimization of clogging during a phacoemulsification operation. These needles operate in a similar manner as the first embodiment of the needle 100 described above, and only the tip portion of the needles are illustrated. It will be understood that the remainder of the proximal end and body of these additional needle is consistent with the structure shown in FIG. 1 or described above. Furthermore, the needle tips of these embodiments may be generally square, circular, flaring, or non-flaring, as shown and described above. The inventors believe that these additional needle embodiments will allow for a more efficient phacoemulsification (reduction of energy used) and a reduced potential for clogging of phacoemulsification compared to prior art needles in general. While these needles may be suitable for a multitude of handpieces, such as longitudinal, torsional, blended handpieces, it is presently believed that the needles are most suited for longitudinally-moving handpieces, such as a longitudinally-vibrating piezoelectric handpiece or a longitudinally-moving pressure type handpiece (such as is shown in United States Patent Application Publication US20180318132 A1, which is incorporated by reference herein in its entirety).")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have Itkowitz in view of Venkataraman to incorporate the teachings of Akahoshi to provide a “18. The method of Claim 1, wherein the operation is a phacoemulsification operation.” Doing so would Eliminate the need for employing expensive fluidics, as recognized by Akahoshi . (Paragraph 62).
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
US 20200075027 to ARANTES discloses receiving a live audio describing an equipment problem and obtaining equipment sensor data in response.
CN 114067985 to SUN discloses collects medical data over a time period. Where the users voice description locates a problem related to the data.
US 20200125084 to Harder discloses receiving equipment data, detecting a fault and providing troubleshooting instructions.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALI M HASSAN whose telephone number is (571)272-5331. The examiner can normally be reached Monday - Friday 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALI M HASSAN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
09/30/2026
Prosecution Insights