DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment and Arguments.
Applicant’s arguments, see page 12, filed 3/16/2026, with respect to claims 1-20 rejection have been fully considered and in light of the current amendments. The 101 rejection of claims 1-20 has been withdrawn.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are not persuasive. Applicant states that “As the Office Action acknowledges, Grantcharov nowhere teaches monitoring a live audio stream during a procedure or detecting, within that live audio stream, an indicator of a problem occurring within a time window of the audio stream. See Office Action at 16 (acknowledging that Grantcharov "does not explicitly teach all of detecting within the audio stream an indicator to a problem, the indicator occurring at least during a time window[.]"). Therefore, Grantcharov fails to teach or suggest at least, "during the operation: detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window," as recited in amended claim 1.”. However, the examiner strongly disagrees see paragraph 102 which teaches live input feeds during a procedure. Therefore, applicants’ argument for claim 1 is not persuasive and rejection is till maintained.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The newly modified claim limitation “detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;” necessitates the new ground of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 19, and 20 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20180122506 A1, (YANG; Kevin Lee.) in view of US 20220253044 A1, (Tremblay; Pallav).
Claim 1, 19, and 20
Regarding Claim 1, 19, and 20, YANG teaches
1. A method comprising:
obtaining during an operation involving a medical system, an audio stream comprising speech by a user using the medical system for performing the operation, wherein the operation is performed upon a subject;
(Paragraph 122 " The platform 10 includes hardware units 20 that include a collection or group of data capture devices for capturing and generating medical or surgical data feeds for provision to encoder 22. The hardware units 20 may include cameras 30 (e.g., wide angle, high definition, pan and zoom camera, such as a Sony EVI-HD1™ or other example camera) mounted within the surgical unit, ICU, emergency unit or clinical intervention units to capture video representations of the OR as video feeds for provision to encoder 22. The video feed may be referred to as medical or surgical data. An example camera 30 is a laparoscopic or procedural view camera (AIDA™, Karl Storz™ or equivalent) resident in the surgical unit, ICU, emergency unit or clinical intervention units. Example video hardware includes a distribution amplifier for signal splitting of Laparoscopic cameras. The hardware units 20 have audio devices 32 (e.g., condenser gooseneck microphones such as ES935ML6™, Audio Technica™ or other example) mounted within the surgical unit, ICU, emergency unit or clinical intervention units to provide audio feeds as another example of medical or surgical data. Example sensors 34 installed or utilized in a surgical unit, ICU, emergency unit or clinical intervention units include but not limited to: environmental sensors (e.g., temperature, moisture, humidity, etc., acoustic sensors (e.g., ambient noise, decibel), electrical sensors (e.g., hall, magnetic, current, mems, capacitive, resistance), flow sensors (e.g., air, fluid, gas) angle/positional/displacement sensors (e.g., gyroscopes, altitude indicator, piezoelectric, photoelectric), and other sensor types (e.g., strain, level sensors, load cells, motion, pressure). The sensors 34 provide sensor data as another example of medical or surgical data. The hardware units 20 also include patient monitoring devices 36 and an instrument lot 18.")
obtaining at least one data stream from equipment associated with the medical system; and during the operation:
(paragraph 293 "A step includes the provisioning of a robust protocol for data extraction into a relational database management system (RDBMS) with a well-defined entity-relationship diagram. The black box data (e.g., data provided to encoder 22 in the form of various data stream feeds) may include video data (in-room videos and the procedural video), audio data (directional microphones and decibel level in room), physiological patient data, signal output data from surgical instruments, room temperature data, and foot traffic data from motion sensing input devices, etc., among others. These feeds may be provided at different times and may require synchronization by endoder 22 prior to extraction. In some embodiments, the feeds already have features extracted and provided in the form of machine-readable and/or interpretable formats, in other embodiments, the feeds may first require processing or pre-processing to extract feature sets."
Paragraph 102 " In another aspect, there is provided a method for collecting and processing medical or surgical data. The method involves receiving, at a multi-channel encoder, a plurality of live or real-time independent input feeds from one or more data capture devices located in an operating room or other patient intervention area, the input feeds relating to a live or real-time medical procedure;")
selecting an audio segment within the live audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data captured, respectively, during at least the time window; [[and]] outputting the audio segment and the at least one data segment,
(paragraph 293 "A step includes the provisioning of a robust protocol for data extraction into a relational database management system (RDBMS) with a well-defined entity-relationship diagram. The black box data (e.g., data provided to encoder 22 in the form of various data stream feeds) may include video data (in-room videos and the procedural video), audio data (directional microphones and decibel level in room), physiological patient data, signal output data from surgical instruments, room temperature data, and foot traffic data from motion sensing input devices, etc., among others. These feeds may be provided at different times and may require synchronization by endoder 22 prior to extraction. In some embodiments, the feeds already have features extracted and provided in the form of machine-readable and/or interpretable formats, in other embodiments, the feeds may first require processing or pre-processing to extract feature sets.
paragraph 350 "For example, relationships may be stored in the form of attribute-relation file format files (ARFF), among others. Experimental data was recorded from 31 cases/surgeries that occurred between May 2015 and August 2015, and audio recordings were analyzed. 559 technical errors (e.g., where a surgeon made an error such as applying too much pressure to a patient's intestine) were tracked, where there were 207 technical events (e.g., where the pressure on a patient's intestine caused the intestine to rupture). Two lengths of recording clips were made, a first set at 30 seconds, and a second set at 30 seconds. Voice recordings were made 5 seconds before error and 25 seconds after error for 30 second recording clips, and made 10 seconds before the error and 50 seconds after the error for 60 second sound recording clips."
Paragraph 79 "The system may have an encoder with a network server for synchronizing and recording the real-time medical or surgical data streams to a common clock or timeline to generate a session container file. As noted, the synchronization may aggregate independent data feeds in a consistent manner to generate a comprehensive data feed generated by data from multiple independent devices.")
outputting the audio segment and the at least one data segment.
(paragraph 266 "The CMMS 1620 may aggregate all media and data (audio, video, device data, sensor data, logs, and so on) and package, compress and encrypt to generate output files. Output files will be organized on network accessible storage devices using standardized file-naming conventions, keywords, file folders, and so on.")
troubleshooting the problem using the audio segment and the at least one data segment and
(paragraph 156 " According to some embodiments, this information may include: video from the procedural field; video of the clinical environment; audio; physiological data from the patient; environmental factors through various sensors (e.g., environmental, acoustic, electrical, flow, angle/positional/displacement and other potential sensors); software data from the medical devices used during intervention; and/or individual data from the healthcare providers (e.g., heart rate, blood pressure, skin conductance, motion and eye tracking, etc.)."
Paragraph 157 "According to some embodiments, this information then may be synchronized (e.g., by the encoder 22) and/or used to evaluate: technical performance of the healthcare providers; non-technical performance of the clinical team members; patient safety (through number of registered errors and/or adverse events); occupational safety; workflow; visual and/or noise distractions; and/or interaction between medical/surgical devices and/or healthcare professionals, etc."
paragraph 282 "Embodiments described herein may provide an application of data-driven modeling to identify, and extract features, correlations and signatures from data collected and analyzed from the OR black box encoder. Data-driven modeling offers a sound perspective to describe and analyze all those systems for which closed-form analytical expressions may be difficult to determine. Using datasets of input-output pairs of samples related to the problem, the objective is to use Computational Intelligence (CI) to reconstruct a mathematical model that recognizes key factors and predicts clinical outcomes, costs and safety hazards. CI tools may include neural networks, support vector machines, fuzzy inference systems, and several techniques from time-series a nalysis and dynamical complex systems. Using CI-based approaches, both offline and online solutions could be built for analyzing errors, adverse events and adverse outcomes in surgery. The term offline refers to solutions that may be used to automatically infer knowledge (e.g., rules of causations, correlations) from examples describing past events recorded in the OR. The online approach may provide a real-time tool to assist surgeons and OR teams intra-operatively. Such an instrument may operate by monitoring the current conditions in the OR, reporting events that may lead to conditions of potential errors (e.g., the noise level, temperature, number of individuals in the room, and so on).")
updating the medical system based on the troubleshooting to correct the problem.
(paragraph 161 " Analyzing these underlying factors according to some embodiments may allow one or more of: (i) proactive monitoring of clinical performance; and/or (ii) monitoring of performance of healthcare technology/devices (iii) creation of educational interventions—e.g., individualized structured feedback (or coaching), simulation-based crisis scenarios, virtual-reality training programs, curricula for certification/re-certification of healthcare practitioners and institutions; and/or identify safety/performance deficiencies of medical/surgical devices and develop recommendations for improvement and/or design of “intelligent” devices and implants—to curb the rate of risk factors in future procedures and/or ultimately to improve patient safety outcomes and clinical costs."
paragraph 177 "Possible applications for some embodiments include one or more of the following: (i) Documentation of various aspects of patient care in clinical areas with a high-risk for adverse outcomes. Comprehensive data collection by the encoder according to some embodiments may enable and/or provide for a detailed reconstruction of any clinical encounter. (ii) Analysis of chains of events leading to adverse outcomes. The data collection and processing according to some embodiments provide an opportunity to retrospectively evaluate one or more mechanisms and/or root causes leading to adverse outcomes in medicine and surgery. (iii) The analysis according to some embodiments may generate knowledge of the incidence and background of human errors and may enable development of strategies to mitigate the consequences of such errors. (iv) Design of training interventions for surgical teams. According to some embodiments, all identified crisis scenarios may be stored in a database and associated with simulation interventions which aim to prepare clinical teams for common clinical challenges and mitigate the impact of errors on clinical outcomes. (v) Evaluation/Improvement/development of existing/new healthcare technology and new treatments. According to some embodiments, the comprehensive data set may be used to evaluate safety hazards associated with implementation of new healthcare technologies. Furthermore, it may enable evaluation of the impact of healthcare technologies on efficiency. (vi) Use for certification and accreditation purposes. According to some embodiments the data may be used for assessment of human performance and development of pass/fail scores using standard setting methodologies."
Paragraph 343 "Mapping procedure complexity and identifying potential hazard zones may be used to create educational strategies targeted directly at these steps. Instructional strategies such as but not limited to deliberate practice may then be used to train for these steps and thus minimize the risk of adverse events. Informing surgeons about complex or hazardous steps also enables the design of standard operating procedures, which is common in aviation (for example the “sterile” cockpit concept during takeoff and landing), to limit distractions during these sensitive steps (no procedure irrelevant conversation, minimize room traffic, reduce overall noise level)."
YANG arguably teaches limitations detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;
However, Tremblay teach
detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;
(paragraph 51 " In some embodiments, the local control system 44 may compare the received audio data to reference audio data (e.g., audio data linked to the industrial automation equipment that is stored in a memory of the local control system 44) to determine whether the audio data is associated with an error in the operation of the industrial automation equipment. For example, the local control system 44 may compare sound characteristics (e.g., volume, pitch, frequency, melody, rhythm, a time period between each note) of the received audio data to sound characteristics corresponding to reference audio data. As another non-limiting example, the sound characteristics may include a time period between each audio tone of the received audio data. As such, the local control system 44 determine that the received audio data does not match the melody when a time period between each adjacent audio tone of the audio tones of the received audio data is different than a reference time period corresponding to the reference audio data. When the sound characteristics of the audio data correlate to or match the sound characteristics of the reference audio data corresponding to the user or other authorized individuals, the local control system 44 may determine that the received audio data does not indicate an error in the operation of the industrial operation components and the local control system 44 may receive additional detected audio data to continue to monitor operation of the industrial automation components. However, if the local control system 44 determines that the received audio data does indicate an error in the operation of the industrial automation equipment, the process 50 may proceed to block 64 and adjust operation of the equipment. In some embodiments, the local control system 44 may output a notification (e.g., an alert or warning) to an operator that indicates that the industrial automation equipment is operating as expected. For example, the notification may be a pop-up notification displayed on a device associated with an individual (e.g., a smart phone, table, and laptop) that indicates the components that are not working as expected. In some embodiments, the notification may indicate a maintenance procedure to fix the component that is not working as expected.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG to incorporate the teachings of Tremblay to provide a “detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;” Doing so would Allow an individual to know of an error without inspecting as recognized by Tremblay. (Paragraph 17).
Regarding claim 19 YANG Further teaches, A computerized apparatus having a processor coupled with a memory unit, the processor being adapted to perform the steps of:
(paragraph 202 "For simplicity only one encoder 22 is shown but system may include more encoders 22 to collect feeds from local or remote data capture devices (of hardware unit 20) and exchange data. The encoders 22 may be the same or different types of computing hardware devices. The encoder 22 has at least one processor, a data storage device (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. The encoder 22 components may be connected in various ways including directly coupled, indirectly coupled via a network, and distributed over a wide geographic area and connected via a network (which may be referred to as “cloud computing”).")
Regarding claim 20 YANG Further teaches A computer program product comprising a non-transitory computer readable medium retaining program instruction, which instructions when read by a processor, cause the processor to perform:
(Paragraph 397 "The technical solution of embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.")
Claims 2, 4-14, and 16 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20180122506 A1, (YANG; Kevin Lee.) in view of US 20220253044 A1 , (Tremblay; Natalie A) in further view of US 20130093829 A1, (Rosenblatt; Peter L)
Claim 2
Regarding Claim 2, YANG in view of Tremblay do not explicitly teach all of the method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream at least one word selected from a predetermined list of words.
However Rosenblatt teaches teach the method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream at least one word selected from a predetermined list of words.
(Paragraph 106 "By way of further example, the systems, and methods may use a device as described herein in a procedure such as a sling procedure. In embodiments, the staff, or a user may choose "TVT-O", a commercially available transobturator sling from Ethicon Women's Health & Urology. In some embodiments, the steps may be included in the program, and in some embodiments, all 27 steps may be listed with short titles, in DVD format. In some embodiments, when the step is chosen, the menu item may be enlarged, and other steps may still be listed, for example on the side of the display. In some embodiments, each step may include a photo, diagram, video, animation, and the like that may demonstrate or provide information about a specific step. In some embodiments, a visual step may be accompanied with text, or subtitles explaining the step, audio narration of the step, both, and/or other features as described herein. By way of example only, and not to limit the invention, the steps of the TVT-O example noted above may comprise: 1. Patient placed in high dorsal lithotomy position; 2. Foley catheter inserted and placed to drainage; 3. Marking pen used to establish landmarks (which may include a description of mark placement); 4. Local anesthesia used to anesthetize inner thigh skin and epithelium of mid urethra and paraurethral areas; 5. Local anesthesia injected into medial portion of obturator foramen; 6. Allis clamps placed 1 cm proximal to external urethral meatus, and at level of bladder neck; 7. 1.5 cm incision made along mid-portion of the urethra; 8. Allis clamp placed on one edge of incision and Metzenbaum scissors used to sharply and bluntly dissect paraurethral space until underside of descending ischiopubic ramus is reached; 9. With Metzenbaum scissors pointed at 45 degree angle and with scissor handle pushed up against contralateral buttocks, firm but controlled pressure on scissors perforates the obturator membrane; 10. Winged guide placed just behind scissors and advanced through obturator membrane; 11. Helical needle placed within winged guide with handle at slight angle (hand position may be displayed); 12. Helical needle pushed laterally though obturator membrane; 13. Winged guide removed; 14. Handle of helical needle dropped vertically while curved needle is brought around ischiopubic ramus and out through medial thigh Skin incision may be made to assist with needle perforation through skin; 15. Needle withdrawn partially while holding on to proximal portion of white needle cover; 16. Kelly clamp grasps white needle cover end; 17. Helical needle completely withdrawn and disposed; 18. White needle cover pulled through skin and Kelly replaced on mesh with plastic sleeve; 19. White needle cover cut off with scissors; 20. Allis clamp on side is replaced on Contralateral side; 21. Procedure repeated on Contralateral side with attention to laying the mesh flat without twists; 22. Cystoscopy performed with 300 cc in the bladder to assure bladder integrity; 23. Foley catheter replaced after cystoscopy and before sling tensioning; 24. Sling tension adjustment performed (e.g. Metzenbaum between mesh and urethra, Babcock technique) and plastic sleeves removed; 25. Vaginal incision closed with running absorbable suture; 26. Sling arms cut just below level of skin; 27. Dermal glue or fine interrupted suture placed on thigh incisions. In embodiments, the surgeon may have the option of viewing a more detailed DFU/IFU, and/or a more detailed explanation of each step. In some embodiments, there may be a set of frequently asked questions and an area to "troubleshoot" based on key words and/or phrases. In embodiments, the surgeon may be able to access immediate technical support through the system for questions or problems with a medical device. In embodiments, reporting mechanisms for complications experienced may also be available. In some embodiments, the device may be a capital expense for hospitals, and/or the data contained on the system may be updatable, and medical device manufactures may have the option to participate in the system for both a set-up charge as well as a subscription price, and the like."
Paragraph 109 "In embodiments, methods and systems of a procedure instruction system may be enabled to log time stamp data about steps undertaken during a medical procedure using a medical device on a patient and associating the time stamp data with the steps undertaken in a work flow for the medical procedure. In some embodiments, the procedure instruction system may be enabled to receive voice commands. A procedure instruction system may log time stamp information based on one or more voice commands received by the procedure instruction system.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG in view of Tremblay to incorporate the teachings of Rosenblatt to provide a “detecting within the live audio stream an indicator to a problem, the indicator occurring at least during a time window;” Doing so would Enable immediate support for the surgeon, as recognized by Rosenblatt. (Paragraph 106).
Claim 4
Regarding Claim 4, further Rosenblatt teach
The method of Claim 1, wherein the audio segment and the at least one data segment comprise, respectively, audio and data captured during a second time window preceding the time window.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 272 "Data/communication/analytics/reporting aspects associated with a procedure instruction system as described herein may include logging/recording. This aspect may further include logging of steps taken by the voice interface. This nature of this aspect might be based on what kind of consent you have regarding the video recording. This aspect may further include video recording/logging for quality assurance that may include addressing legal discovery standards, comparing a library of successful procedures, reviewing for mistakes, and the like. This aspect may further include documenting the number of times a doctor used a particular procedure (e.g. for medical boards and societies), logging feedback from a surgeon, logging the timeout activities, and the like. This aspect may further include recording of additional equipment used that may include beginning log/record process upon taking equipment out of a box, automatically logging unboxing events, and the like. This aspect may further include timing the procedure. The procedure may include a number of steps. This aspect may further include logging time-stamped information about each step undertaken during the procedure by the professional using the medical device on the patient. This aspect may further include storing images with directions."
Paragraph 110 "The time stamp data may include logged information, such as, but not limited to a start time stamp for the medical procedure, a start time stamp for each step in the medical procedure, an end time stamp for each step in the medical procedure, an end time stamp for the medical procedure, a start time stamp for an interaction with a specialist during the medical procedure, an end time stamp for an interaction with a specialist during the medical, and other logged information with a time stamp.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG to incorporate the teachings of Rosenblatt to provide a “The method of Claim 1, wherein the audio segment and the at least one data segment comprise, respectively, audio and data captured during a second time window preceding the time window.” Doing so would ensure effective completion of procedures, as recognized by Rosenblatt. (Paragraph 5).
Claim 5
Regarding Claim 5, Rosenblatt further teaches
. The method of Claim 4, wherein the second time window starts at a predetermined event.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 110 "The time stamp data may include logged information, such as, but not limited to a start time stamp for the medical procedure, a start time stamp for each step in the medical procedure, an end time stamp for each step in the medical procedure, an end time stamp for the medical procedure, a start time stamp for an interaction with a specialist during the medical procedure, an end time stamp for an interaction with a specialist during the medical, and other logged information with a time stamp.")
See claim four for rationale.
Claim 6
Regarding Claim 6, Rosenblatt further teaches
The method of Claim 4, wherein the second time window starts at a beginning time of the operation.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 110 "The time stamp data may include logged information, such as, but not limited to a start time stamp for the medical procedure, a start time stamp for each step in the medical procedure, an end time stamp for each step in the medical procedure, an end time stamp for the medical procedure, a start time stamp for an interaction with a specialist during the medical procedure, an end time stamp for an interaction with a specialist during the medical, and other logged information with a time stamp.")
See claim four for rationale.
Claim 7
Regarding Claim 7, Rosenblatt further teach
The method of Claim 1, wherein the audio segment and the at least one data segment comprise, respectively, audio and data captured during a third time window consecutive to the time window.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 158 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems for scheduling a plurality of resources that may comprise logging times for steps taken in a step-by-step workflow associated with a procedure and a resource, projecting the completion time for the workflow, and generating a schedule for the resource based on the projected completion time for the workflow. By way of example, a practitioner may utilize an instruction system wherein a tablet device may be used to log time stamp data and/or other data associated with a procedure, such as performing an EKG. As the practitioner completes each step, the completion time may be logged and a projected time for the EKG machine to be available may be computed such that schedule may be generated based on said completion time.")
See claim four for rationale.
Claim 8
Regarding Claim 8, Rosenblatt further teach
The method of Claim 7, wherein the third time window ends at a predetermined event or at an end time of the operation.
(Paragraph 157 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems of a first procedure instruction system that may include notifying one or more users of a second procedure instruction system that a host practitioner using the first procedure instruction system has almost completed the steps in a workflow and beginning a subsequent workflow by the user(s) of the second procedure instruction system. By way of example, a practitioner may use said first procedure instruction system for guidance when using a medical device and the system may gather data related to the time the practitioner has logged on each step, and it may notify one or more users of a second system that the practitioner has completed a particular step in the workflow such that those associated with the second system may provide support or preparation for next steps in one or more workflows."
Paragraph 158 "Methods and systems may guide a practitioner host to use a medical device in a procedure environment where the procedure instruction system may be comprised of a tablet device, a system for logging time stamp data about the steps undertaken using a medical device and a communication system. Such methods and systems may be associated with methods and systems for scheduling a plurality of resources that may comprise logging times for steps taken in a step-by-step workflow associated with a procedure and a resource, projecting the completion time for the workflow, and generating a schedule for the resource based on the projected completion time for the workflow. By way of example, a practitioner may utilize an instruction system wherein a tablet device may be used to log time stamp data and/or other data associated with a procedure, such as performing an EKG. As the practitioner completes each step, the completion time may be logged and a projected time for the EKG machine to be available may be computed such that schedule may be generated based on said completion time.")
See claim four for rationale.
Claim 9
Regarding Claim 9, YANG further teaches
The method of Claim 1, further comprising concealing identifying details of the subject or of the user within the audio segment and the at least one data segment.
(Paragraph 127 "Anonymization Software for anonymizing and protecting the identity of all medical professionals, patients, distinguishing objects or features in a medical, clinical or emergency unit. This software implements methods and techniques to detect facial, distinguishing objects, or features in a medical, clinical or emergency unit and distort/blur the image of the distinguishing element. The extent of the distortion/blur is limited to a localized area, frame by frame, to the point where identity is protected without limiting the quality of the analytics."
Paragraph 128 "Voice or Vocabulary Alteration Software for anonymizing and protecting the identity of all medical professionals, patients, distinguishing objects or features in a medical, clinical or emergency environment. This software may implement methods and techniques running on hardware in a medical, clinical or emergency environment to alter voices, conversations and/or remove statements of everyday language to preserve the identity of the speaker while at the same time maintaining the integrity of the input stream so as to not adversely impact the quality of the analytics."
Paragraph 147 "In another aspect, embodiments described herein may involve an encoder configured for hosting and operating anonymization and voice or vocabulary alteration software(s) for the purpose of protecting the identity of medical professionals, patients, distinguishing objects or features in a medical, clinical or emergency environment. This may be done either before compressing, containerizing and/or encrypting the collective data, or after receipt of transmission to back office and decryption."
Paragraph 194 "According to some embodiments, hardware units 20 may have patient monitor devices 36 (FIG. 1). For example, patient monitor devices 35 may include an anesthesia machine monitor that may be used to observe physiological data of the patient in real-time and to detect abnormal changes in patient vital signs. According to some embodiments, the vital sign display may be extracted from the anesthesia machine using a video card, which generates a secondary feed of VGA output. The vital sign video feed may be converted from VGA to HD-SDI format using a converter unit (VidBlox 3G-SL, PESA, Huntsville, Ala., USA), prior to integration and synchronization with the other video feeds."
Paragraph 195 " In some embodiments, there may be extraction of raw digital data from the anesthesia device directly for provision to encoder 22 which ingests it as metadata.")
Claim 10
Regarding Claim 10, YANG further teaches
The method of Claim 1, wherein the at least one data stream comprises data related to actions performed by the user.
(Paragraph 274 "Embodiments described herein may implement motion tracking using 3D cameras or IR devices. For example, the black box platform may collect and ingest motion tracking data for people and objects at the surgical site. To maintain complete freedom in a clinical environment, markerless motion tracking may be required. Data may be collected from 3D cameras or time-of-flight cameras/sensors."
Paragraph 293 "A step includes the provisioning of a robust protocol for data extraction into a relational database management system (RDBMS) with a well-defined entity-relationship diagram. The black box data (e.g., data provided to encoder 22 in the form of various data stream feeds) may include video data (in-room videos and the procedural video), audio data (directional microphones and decibel level in room), physiological patient data, signal output data from surgical instruments, room temperature data, and foot traffic data from motion sensing input devices, etc., among others. These feeds may be provided at different times and may require synchronization by endoder 22 prior to extraction. In some embodiments, the feeds already have features extracted and provided in the form of machine-readable and/or interpretable formats, in other embodiments, the feeds may first require processing or pre-processing to extract feature sets.")
Claim 11
Regarding Claim 11, YANG further teaches
The method of Claim 10, wherein the actions performed by the user comprise user interface activation actions.
(Paragraph 217 "According to an embodiment, a user interface may be provided on a PC-based touchscreen monitor. The user interface may be referred herein as a Control Interface 14 (FIG. 1) and may serve as a “central control” station that records video and audio feeds in real-time, and transmits control commands to the encoder 22. The Graphical User Interface (GUI) and its parameters may incorporate principles of UI design to provide an interface is simple user-friendly and functional."
Paragraph 218 "According to an embodiment, the features of the Control Interface 14 providing the central control station (e.g., computer, tablet, PDA, hybrid, convertible) may be located in the clinical unit or another customer designated location. It contains a customizable graphical user interface (GUI) that provides a simple, user friendly and functional control of the system.")
Claim 12
Regarding Claim 1, YANG further teaches
The method of Claim 1, wherein the at least one data stream comprises data related to actions taken by the system.
(Paragraph 194 "According to some embodiments, hardware units 20 may have patient monitor devices 36 (FIG. 1). For example, patient monitor devices 35 may include an anesthesia machine monitor that may be used to observe physiological data of the patient in real-time and to detect abnormal changes in patient vital signs. According to some embodiments, the vital sign display may be extracted from the anesthesia machine using a video card, which generates a secondary feed of VGA output. The vital sign video feed may be converted from VGA to HD-SDI format using a converter unit (VidBlox 3G-SL, PESA, Huntsville, Ala., USA), prior to integration and synchronization with the other video feeds.")
Claim 13
Regarding Claim 13, Rosenblatt further teach
The method of Claim 12, wherein the actions are taken by the system in response to user actions.
(Paragraph 301 "Use scenarios and applications associated with a procedure instruction system as described herein may further include parameter measurement/evaluation and feedback during a procedure. This aspect may further include parameters that may include size, configuration, surface characteristics, temperature, appearance, measured parameter (such as for example camera or other inspection fed to person guiding procedure), and the like. This aspect may further include guidance based on feedback such as choice of step, duration of step, omission of step, use/non-use of optional device feature(s), and the like. This aspect may further include the guidance based on information from other procedures using the medical device, information from procedures using other medical devices, information from procedures by the steps undertaken by other professionals, information from other procedures by the medical professional, and the like. This aspect may further include using a Kinect-like camera to monitor motion, location, position, angle, etc. of device to generate feedback to take a corrective action. (e.g. "Your thumb is in the wrong place, move it here;" "the angle of the device is wrong, tilt it this way;" "you are touching the wrong tissue, move this way," etc.).")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG to incorporate the teachings of Rosenblatt to provide a “13. The method of Claim 12, wherein the actions are taken by the system in response to user actions.” Doing so would Ensure the medical professional is following the proper guidance in an operation, as recognized by Rosenblatt. (Paragraph 301).
Claim 14
Regarding Claim 14, Rosenblatt further teach
The method of Claim 13, wherein the at least one data stream comprises data related to measurements taken by the system.
(Paragraph 301 "Use scenarios and applications associated with a procedure instruction system as described herein may further include parameter measurement/evaluation and feedback during a procedure. This aspect may further include parameters that may include size, configuration, surface characteristics, temperature, appearance, measured parameter (such as for example camera or other inspection fed to person guiding procedure), and the like. This aspect may further include guidance based on feedback such as choice of step, duration of step, omission of step, use/non-use of optional device feature(s), and the like. This aspect may further include the guidance based on information from other procedures using the medical device, information from procedures using other medical devices, information from procedures by the steps undertaken by other professionals, information from other procedures by the medical professional, and the like. This aspect may further include using a Kinect-like camera to monitor motion, location, position, angle, etc. of device to generate feedback to take a corrective action. (e.g. "Your thumb is in the wrong place, move it here;" "the angle of the device is wrong, tilt it this way;" "you are touching the wrong tissue, move this way," etc.).")
See claim 13 for rationale.
Claim 16
Regarding Claim 16, YANG further teaches
The method of Claim 1, wherein the at least one data stream comprises at least one type of data selected from the group consisting of: raw Electrocardiogram (ECG) digital sample recordings collected from ECG electrodes; ECG processed data; Intra-cardiac catheter navigation recordings; Intra-cardiac catheter shaft visualization; catheter localization and visualization algorithmic computation results; localization based on electromagnetic signals; localization based on currents; therapeutic radio frequency ablation or pulse field ablation applications including time duration or repetition information recordings; recording of the measurement of force applied; user action log, including recording of actions performed by the user with a user interface module; keyboard, mouse, or touch screen actions; and ultrasound frames or anatomical markers.
(Paragraph 217 "According to an embodiment, a user interface may be provided on a PC-based touchscreen monitor. The user interface may be referred herein as a Control Interface 14 (FIG. 1) and may serve as a “central control” station that records video and audio feeds in real-time, and transmits control commands to the encoder 22. The Graphical User Interface (GUI) and its parameters may incorporate principles of UI design to provide an interface is simple user-friendly and functional."
Paragraph 218 "According to an embodiment, the features of the Control Interface 14 providing the central control station (e.g., computer, tablet, PDA, hybrid, convertible) may be located in the clinical unit or another customer designated location. It contains a customizable graphical user interface (GUI) that provides a simple, user friendly and functional control of the system.
paragraph 219 "According to an embodiment, the Control Interface 14 may have a Play/Pause button. Some segments of the procedure may not need to be recorded. To skip these segments from the recording, the user interface may pause and restart the recordings when desired by way of control commands generated in response to activation of the play/pause button. The pause and play time-stamps may be recorded in a log file, indicating the exact times of the procedure that were extracted.")
Claims 3 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20180122506 A1, (YANG; Kevin Lee.) in view US 20220253044 A1 , (Tremblay; Natalie A) in further view of US Patent US 20230185524 A1, (KOKER; Ekin).
Claim 3
Regarding Claim 3 YANG in view of Tremblay teach do not explicitly teach all of 3. The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.
However, KOKER teach
The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.
(Paragraph 77 "Likewise, the operation 212 analyzes the audio of the voice of the health professional to determine actionable information about the medical professional. For example, voice analysis of the medical professional’s voice may be applied to detect signs of tiredness, anger, depression, or so forth, possibly indicative of burnout, disinterest and boredom which may indicate a need for greater challenges, diversity of cases, opportunities for growth and development, signs of continuous or constant stress may indicate a need for additional training, and so forth. In another example, speech recognition (i.e., NLP) can be used to identify issues such as detecting abusive, inappropriate language, abrasive behaviors towards patients or fellow staff members, recognize voice commands to trigger calls to other patients (i.e., a radiology, remote expert, or so forth), initiate emergency protocols when help is needed (call 911, alert security, and so forth), recognize workflow stages based on words or phrases detected (e.g., “last scan,” “five minutes left in the examination, etc.), create timelines based on recognition of workflow activities, and so forth. As one illustrative example, voice analytics may be performed on speech of the medical professional in the audio to detect stress or exhaustion of the medical professional. In another example, the voice analytics may detect slurred speech of the medical professional indicative of impairment by alcohol or drugs. Again, these are merely nonlimiting illustrative examples.")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG in view of Tremblay to incorporate the teachings of KOKER to provide a “3. The method of Claim 1, wherein the indicator to the problem is detected by recognizing within the audio stream an emotional state indicative of a problem.” Doing so would Detect abusive doctors, as well as impaired doctors , as recognized by KOKER. (Paragraph 77).
Claims 15 and 17 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20180122506 A1, (YANG; Kevin Lee.) in view of US 20220253044 A1 , (Tremblay; Natalie A) in further view of US Patent US 20230225800 A1 , (Grund; Alessa Tabea ).
Claim 15 and 17
Regarding Claim 15 and 17 YANG in view of Tremblay teach do not explicitly teach all of the method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.
However, Grund teach
The method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.
(paragraph 20 "FIG. 1A shows one embodiment of a combined cardiac electrophysiological mapping (EP), pacing and ablation system 100;
Some Embodiments of Electrophysiological Mapping and Pacing and Ablation Systems, Computer Systems, Intracardiac Imaging and Navigation Systems, and Body Surface Imaging and Navigation Systems")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG in view of Tremblay to incorporate the teachings of Grund to provide a “The method of Claim 1, wherein the system is an electrophysiological (EP) mapping and ablation system.” Doing so would Result in higher rates of success in treating cardiac rhythm disorders, as recognized by Grund . (Paragraph 3).
Claims 18 are rejected under 35 U.S.C. 103 as obvious over US Patent US 20180122506 A1, (YANG; Kevin Lee.) in view of US 20220253044 A1 , (Tremblay; Natalie A) in further view of US Patent US 20220370246 A1 , (Akahoshi; Takayuki ).
Claim 18
Regarding Claim 18 YANG in view of Tremblay teach do not explicitly teach all of 18. The method of Claim 1, wherein the operation is a phacoemulsification operation.
However, Akahoshi teach
The method of Claim 1, wherein the operation is a phacoemulsification operation.
(Paragraph 83 "FIGS. 23-28 illustrate additional embodiments of phacoemulsification needles according to the present invention containing internal elements or projections within the tip portion of the needle for improving the efficiency of the needle and minimization of clogging during a phacoemulsification operation. These needles operate in a similar manner as the first embodiment of the needle 100 described above, and only the tip portion of the needles are illustrated. It will be understood that the remainder of the proximal end and body of these additional needle is consistent with the structure shown in FIG. 1 or described above. Furthermore, the needle tips of these embodiments may be generally square, circular, flaring, or non-flaring, as shown and described above. The inventors believe that these additional needle embodiments will allow for a more efficient phacoemulsification (reduction of energy used) and a reduced potential for clogging of phacoemulsification compared to prior art needles in general. While these needles may be suitable for a multitude of handpieces, such as longitudinal, torsional, blended handpieces, it is presently believed that the needles are most suited for longitudinally-moving handpieces, such as a longitudinally-vibrating piezoelectric handpiece or a longitudinally-moving pressure type handpiece (such as is shown in United States Patent Application Publication US20180318132 A1, which is incorporated by reference herein in its entirety).")
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified YANG in view of Tremblay to incorporate the teachings of Akahoshi to provide a “18. The method of Claim 1, wherein the operation is a phacoemulsification operation.” Doing so would Eliminate the need for employing expensive fluidics, as recognized by Akahoshi . (Paragraph 62).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALI M HASSAN whose telephone number is (571)272-5331. The examiner can normally be reached Monday - Friday 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALI M HASSAN/Examiner, Art Unit 2653 /Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
06/22/2026
Prosecution Insights