Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The following is in response to the amendment filed on March 31, 2026.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-13 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc et al (US PG Pub 2019/00909069) in view of Wolf et al (US Patent 11/426255) and further in view of King et al (“Hand Gesture Recognition With Body Sensor Networks”).
With respect to claim 1:
Jarc teaches:
A system comprising:
A processor, the processor configured to:
Acquire image data from a camera targeting a location where healthcare is administered; (Paragraph [043], discloses a camera mounted with an endoscope that captures images of a surgical site)
Receive sensor data from a sensor during performance of a multi-step medical procedure; (Paragraph [090], discloses receiving sensor data during a surgical procedure)
Generate gesture features based on the sensor data by applying the sensor data as input to a trained gesture recognition model configured to classify physical movements of the care provider; (Paragraph [091] and [094], disclose generating a define gestured based on inputting the sensor data to the gesture assessor and classifying the gestures using a neural network)
Generate image features by applying the image data as input to a trained image feature recognition model; determine based on the image feature and a trained procedure model configured to decode the image features a step identifier for a step in a multi-step surgical procedure; (Paragraphs [093] and [094], disclose a task assessor that uses the images and a neural network to determine a current task (surgery) step being performed)
Determine, based on the gesture features and a trained performance model a performance metric that measures performance of the care provider; and (Paragraphs [113] and [117], disclose using the assessments determine by the gesture assessor and a neural network to produce a score for the quality of the surgical performance)
Output the performance metric, where in to output the performance metric the processor is configured to: store the performance metric in a memory, communicate the performance metric over a communications network, display the performance metric on the display, or a combination thereof. (Paragraph [122], discloses storing the scored for the surgical performance for further analysis)
Jarc does not appear to explicitly disclose:
A sensor attached to a care provider;
Access natural language text descriptive of an instruction to perform the step;
Display the natural language text associated with the step on a display accessible to the care provider;
Wolf teaches:
Access natural language text descriptive of an instruction to perform the step; (Col 158 Lines 61-67 & Col 159 Lines 1-5 “In some embodiments, a method for providing decision support for surgical procedures may include outputting a recommendation to a user to undertake and/or to avoid a specific action. Such a recommendation may include any guidance, regardless of the form of the guidance (e.g., audio, video, text-based, control commands to a surgical robot, or other data transmission that provides advice and/or direction). In some instances, the guidance may be in the form of an instruction, in others it may be in the form of a recommendation. The trigger for such guidance may be a determined existence of a decision-making junction and an accessed correlation.””
Col 162 Lines 16-21 “In some embodiments, outputting a recommendation may include presenting a first instruction to perform a first step, receiving an indication of that a first step was performed successfully, and, in response to the received indication that a first step was performed successfully, presenting a second instruction to perform a second step.
Examiner’s Note: In order to output a text-based recommendation, the natural language text must be accessed.)
display the natural language text associated with the step on a display accessible to the care provider; (Col 158 Lines 61-67 & Col 159 Lines 1-17 “In some embodiments, a method for providing decision support for surgical procedures may include outputting a recommendation to a user to undertake and/or to avoid a specific action. Such a recommendation may include any guidance, regardless of the form of the guidance (e.g., audio, video, text-based, control commands to a surgical robot, or other data transmission that provides advice and/or direction). In some instances, the guidance may be in the form of an instruction, in others it may be in the form of a recommendation. The trigger for such guidance may be a determined existence of a decision-making junction and an accessed correlation. Outputting a recommendation may include transmitting a recommendation to a device, displaying a recommendation on an interface, and/or any other mechanism for supplying information to a decision maker. Outputting a recommendation to a user may include outputting a recommendation to a person in an operating room, to a surgeon (e.g., a human surgeon and/or a surgical robot), to a person assisting a surgical procedure (e.g., a nurse), and/or any to other user. For example, outputting a recommendation may include transmitting a recommendation to a computer, a mobile device, an external device, smart glasses, a projector, a surgical robot, and/or any other device capable of conveying information to the user.)
Jarc and Wolf are considered to be analogous to each other as they are in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with the access and display of the natural language text instructions for performing certain steps taught by Wolf in order to provide surgeons with analytical support to prepare for surgeries or receive help during surgeries for positive postoperative results. (Col 1 Lines 29-36)
The combination of Jarc and Wolf does not appear to explicitly disclose:
A sensor attached to a care provider;
King teaches:
A sensor attached to a care provider; (Abstract, teaches a sensor being attached to a surgeon’s hand)
King is analogous to Jarc and Wolf because they are all related to surgical improvements. It would have been obvious to combine the teachings of Jarc, Wolf and King because this would allow for a quantitative assessment of surgical skills being used for surgical training. (King, Abstract)
Regarding Claim 2:
Jarc fails to teach: The system of claim 1, wherein the processor is further configured to: playback an audible explanation of description of how to perform the step.
However, Wolf teaches: The system of claim 1, wherein the processor is further configured to: playback an audible explanation of description of how to perform the step. (Col 158 Lines 61-67 & Col 159 Lines 1-3 “In some embodiments, a method for providing decision support for surgical procedures may include outputting a recommendation to a user to undertake and/or to avoid a specific action. Such a recommendation may include any guidance, regardless of the form of the guidance (e.g., audio, video, text-based, control commands to a surgical robot, or other data transmission that provides advice and/or direction). In some instances, the guidance may be in the form of an instruction, in others it may be in the form of a recommendation.)
Jarc and Wolf are considered to be analogous to each other as they are in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with the access and display of the natural language text instructions for performing certain steps taught by Wolf in order to provide surgeons with analytical support to prepare for surgeries or receive help during surgeries for positive postoperative results. (Col 1 Lines 29-36 “When preparing for a surgical procedure, it may be beneficial for a surgeon to view video footage depicting certain surgical events, including events that may have certain characteristics. In addition, during a surgical procedure, it may be helpful to capture and analyze videos to provide various types of decision support to surgeons. Further, it may be helpful analyze surgical videos to facilitate postoperative activity.”)
Regarding Claim 3:
Jarc further teaches: The system of claim 1, wherein the processor is further configured to: communicate a live video feed of the medical procedure to a device at a remote location, the device configured to receive input information provided by a remote user remotely viewing the medical procedure, the input information comprising audio, images, text, lines, or a combination thereof. ([0067] “In one aspect, the surgical site video captured by the imaging device associated with TSS 850 is recorded by the TSS 850, and stored in database 830, in addition to being displayed in real time or near real time to a user. Highlights and/or annotations associated with the recorded video that were made by the user can also be stored on database 830. In one aspect, the highlights made by the user are embedded with the recorded video prior to its storage on database 830. At a later time, the recorded video can be retrieved for viewing. Examiner’s Note: The highlights and annotations are text and lines which are stored in combination with the video footage.)
Regarding Claim 5:
Jarc further teaches: The system of claim 1, wherein the processor is further configured to: determine respective performance metrics for the steps in the multi-step procedure; ([0117] “Assessment criteria 1406 defines one or more criteria for computing a performance assessment of the surgeon. In one type of embodiment, a performance of a surgical procedure is given a series of scores corresponding to the different identified stages of the procedure. Assessment criteria 1406 may include filtering, masking, and weighting information to be applied to the various metrics in order to adjust their relative importance.
and determine, based on aggregation of the respective performance metrics, a combined performance metric for the multi-step procedure. ([0117] “Notably, different weights/adjustments may be applied to the metrics for different surgical procedures or for different stages of a given surgical procedure. Also, assessment criteria 1406 may define an algorithm or formula(s) to be applied in rendering the surgical performance assessment. In an embodiment where surgical technique assessor 1402 uses a machine-learning algorithm, such as an ANN, classifier, clustering agent, support vector machine, or the like, assessment criteria 1406 may include criteria for constructing feature vectors of metrics.” Examiner’s Note: The performance assessment is the combined performance metric with all weights and algorithms applied to each individual score for each stage of the procedure.)
Regarding Claim 6:
Jarc further teaches: The system of claim 1, wherein the sensor data comprises a measurement of directional movement information, rotation information, vibration information, position information, or a combination thereof, corresponding to the care provider, or an appendage of the care provider, operating in the location where healthcare is administered. ([0116] “In an embodiment, surgical technique assessor 1402 computes various statistics for each of a variety of temporal and spatial metrics... An economy-of-motion analysis may also be performed, which examines a quantity of gestures or other movements, quantity of events, or some combination of these quantities, associated with completion of certain tasks or surgical segments. Idle time (i.e., the absence of movement) may also be taken into account. Other metrics, such as velocity of motion, input-device wrist angles, idle time (e.g., stationarity), master workspace range, master clutch count, applied energy, and other such parameters, may also be taken into account in assessment of surgical technique. Camera control metrics, such as camera movement frequency, camera movement duration, camera movement interval also may be taken into account in assessment of surgical technique.”)
Regarding Claim 7:
Jarc further teaches: The system of claim 1, wherein the processor is further configured to: determine the performance metric satisfies an intervention criterion; ([0130] “Assist criteria 1603 includes various threshold conditions that, when met, trigger the operation of assistive action by one or more of the assistance-rendering engines described below. Notably, different thresholds may be associated with different types of assistance. In operation, procedure performance sensor 1602 compares the current running surgical technique assessment for the current surgical procedure and segment thereof, against a threshold condition corresponding to the procedure and stage. If the threshold condition comparison is met—i.e., if the current running surgical technique assessment score falls below any given threshold, corresponding assistance may be offered and/or provided.”)
and establish, in response to satisfaction of the intervention criterion, communication with a remote node at a different geographic location from where healthcare is administered. ([0103] “If the threshold condition comparison is met—i.e., if the current running surgical technique assessment score falls below any given threshold, corresponding assistance may be offered and/or provided.”
[0132] “Additionally, for example, the assist criteria 1603 could be used to selectively request input from particular expert surgeons with expertise in particular step of a procedure. For example, certain surgeons may be really good at complicated dissections and therefore they should be the ones “on call” if someone struggles with that step whereas other surgeons can be called for simpler steps.”
[0136] “Notifier 1610 is configured to initiate communications with an on-call surgeon according to
expert contact info 1612, who, via a remote console or other interface, can provide direct guidance
and/or assistance to the surgeon.”
[0137] “Technique advisor 1614 may recommend a particular surgical technique to be used in a given scenario, in response to the current or upcoming surgical stage and taking into account the assessed running surgical technique assessment of the surgeon, based on surgical skill requirements database 1616. In a related embodiment, technique advisor 1614 may provide applicable prompting or other instructions or recommendations to other medical personnel, such as to one or more assistants in the surgical environment.”
Examiner’s Note: To clarify, the on-call doctor is called based on the running surgical assessment; this can be affirmed based on Fig. 20. The on-call doctor is read to be contacted and providing guidance via a console in a remote location.
Regarding Claim 8:
Jarc fails to teach: The system of claim 7, further comprising: transmit the performance metric, the image data, the gesture data, or a combination thereof to the remote node.
However, Wolf further teaches: The system of claim 7, further comprising: transmit the performance metric, the image data, the gesture data, or a combination thereof to the remote node. (Col 155 Lines 21-32 "Receiving video footage may occur via a sensor (e.g., an image sensor above a patient, within a patient, or located elsewhere within an operating room), a surgical robot, a camera, a mobile device, an external device using a communication device, a shared memory, and/or any other connected hardware and/or software component capable of capturing and/or transmitting images. Video footage may be received via a network and/or directly from a device via a wired and/or wireless connection. Receiving video footage may include reading, retrieving, and/or otherwise accessing video footage from data storage, such as a database, a disk, a memory, a remote system, an online data storage, and/or any location or medium where information may be retained.” Examiner’s Note: The video footage of the operating site being received to a remote system is read as image data being transmitted to a remote node.)
Jarc and Wolf are considered to be analogous to each other as they are in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with the transmitting of image data to a remote node taught by Wolf in order to fulfill user requests and share image data with multiple users across different locations. (Col 59 Lines 5-19 “For example, the aggregate of the first group of frames may be transmitted through a network to a computing device of the user. As another example, the location of the aggregate of the first group of frames may be shared with the user… For example, a user may request to view a plurality of intraoperative surgical events in the particular surgical footage.”)
Regarding Claim 9:
Jarc further teaches: The system of claim 1, the processor is further configured to: determine the performance metric satisfies an intervention criterion; ([0130] “Assist criteria 1603 includes various threshold conditions that, when met, trigger the operation of assistive action by one or more of the assistance-rendering engines described below… In operation, procedure performance sensor 1602 compares the current running surgical technique assessment for the current surgical procedure and segment thereof, against a threshold condition corresponding to the procedure and stage. If the threshold condition comparison is met—i.e., if the current running surgical technique assessment score falls below any given threshold, corresponding assistance may be offered and/or provided.”)
Jarc fails to teach: in response to satisfaction of the intervention criterion: acquire a second natural language text specifying an instruction to improve the performance of the step; and display the second natural language text and/or playback the natural language text in audio format.
However, Wolf teaches: in response to satisfaction of the intervention criterion: acquire a second natural language text specifying an instruction to improve the performance of the step; (Col 162 Lines 16-21 “In some embodiments, outputting a recommendation may include presenting a first instruction to perform a first step, receiving an indication of that a first step was performed successfully, and, in response to the received indication that a first step was performed successfully, presenting a second instruction to perform a second step.”
Col 160 Lines 26-45 “Consistent with the present embodiments, a recommendation may include a description of a current surgical situation, guidance, an indication of preemptive or corrective measures, an indication of alternative approaches, danger zone mapping, and/or any other information that might inform the surgeon relative to a surgical procedure… A corrective measure may include an action that may improve an outcome.”)
and display the second natural language text and/or playback the natural language text in audio format. (Col 158 Lines 61-67 & Col 159 Lines 1-17 “In some embodiments, a method for providing decision support for surgical procedures may include outputting a recommendation to a user to undertake and/or to avoid a specific action. Such a recommendation may include any guidance, regardless of the form of the guidance (e.g., audio, video, text-based, control commands to a surgical robot, or other data transmission that provides advice and/or direction). In some instances, the guidance may be in the form of an instruction, in others it may be in the form of a recommendation. The trigger for such guidance may be a determined existence of a decision-making junction and an accessed correlation. Outputting a recommendation may include transmitting a recommendation to a device, displaying a recommendation on an interface, and/or any other mechanism for supplying information to a decision maker. Outputting a recommendation to a user may include outputting a recommendation to a person in an operating room, to a surgeon (e.g., a human surgeon and/or a surgical robot), to a person assisting a surgical procedure (e.g., a nurse), and/or any to other user. For example, outputting a recommendation may include transmitting a recommendation to a computer, a mobile device, an external device, smart glasses, a projector, a surgical robot, and/or any other device capable of conveying information to the user.)
Jarc and Wolf are considered to be analogous to each other as they are in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with the access and display of the natural language text instructions for performing certain steps taught by Wolf in order to provide surgeons with analytical support to prepare for surgeries or receive help during surgeries for positive postoperative results. (Col 1 Lines 29-36 “When preparing for a surgical procedure, it may be beneficial for a surgeon to view video footage depicting certain surgical events, including events that may have certain characteristics. In addition, during a surgical procedure, it may be helpful to capture and analyze videos to provide various types of decision support to surgeons. Further, it may be helpful analyze surgical videos to facilitate postoperative activity.”)
Regarding Claim 10:
Jarc fails to teach: The system of claim 1, wherein to display the natural language text, the processor is further configured to: display the natural language text and live video of the location where healthcare is administered on a display device viewable by the care provider.
However, Wolf teaches: The system of claim 1, wherein to display the natural language text, the processor is further configured to: display the natural language text and live video of the location where healthcare is administered on a display device viewable by the care provider. (Col 158 Lines 61-67 & Col 159 Lines 1-17 “In some embodiments, a method for providing decision support for surgical procedures may include outputting a recommendation to a user to undertake and/or to avoid a specific action. Such a recommendation may include any guidance, regardless of the form of the guidance (e.g., audio, video, text-based, control commands to a surgical robot, or other data transmission that provides advice and/or direction). In some instances, the guidance may be in the form of an instruction, in others it may be in the form of a recommendation. The trigger for such guidance may be a determined existence of a decision-making junction and an accessed correlation. Outputting a recommendation may include transmitting a recommendation to a device, displaying a recommendation on an interface, and/or any other mechanism for supplying information to a decision maker. Outputting a recommendation to a user may include outputting a recommendation to a person in an operating room, to a surgeon (e.g., a human surgeon and/or a surgical robot), to a person assisting a surgical procedure (e.g., a nurse), and/or any to other user. For example, outputting a recommendation may include transmitting a recommendation to a computer, a mobile device, an external device, smart glasses, a projector, a surgical robot, and/or any other device capable of conveying information to the user.)
Jarc and Wolf are considered to be analogous to each other as they are in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with the access and display of the natural language text instructions for performing certain steps taught by Wolf in order to provide surgeons with analytical support to prepare for surgeries or receive help during surgeries for positive postoperative results. (Col 1 Lines 29-36 “When preparing for a surgical procedure, it may be beneficial for a surgeon to view video footage depicting certain surgical events, including events that may have certain characteristics. In addition, during a surgical procedure, it may be helpful to capture and analyze videos to provide various types of decision support to surgeons. Further, it may be helpful analyze surgical videos to facilitate postoperative activity.”)
Claims 11-13 and 15-20 are rejected according to claims 1-3 and 5-10 above.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Jarc in view of Wolf in further view of King et al (“Hand Gesture Recognition With Body Sensor Networks”) and further in view of Qiu et al. (US 20200367970 A1).
Regarding Claim 4:
The combination of Jarc, Wolf and King fails to teach: wherein the processor is further configured to: communicate the natural language text to an augmented reality device viewable by the care provider to display input information provided by the remote user.
However, Qiu teaches: The system of claim 1, wherein the processor is further configured to: communicate the natural language text to an augmented reality device viewable by the care provider to display input information provided by the remote user. ([0318] “Moderated walkthroughs combined with traditional education media can be used to enhance education of early-stage learners. These include visualization of a data model from a model sets record 154 fused or embedded to physical teaching models or textbook illustrations. Annotations and comments can be made by the individual learner or classmates, where annotations may be textual notes, audio, or video demonstrating or summarizing the sequence and steps of the procedure.”
[0320] “In at least one embodiment, the client device 170, having a processor, can carry out a method for performing AR-assisted surgical procedure walkthrough, the method comprising: receiving a virtual surgical plan at the client device; …” Examiner’s Note: To clarify, the client device is the AR device. Annotations and comments made by the learner is read as input information provided by the remote user.)
Jarc and Qiu are considered to be analogous to each other as they are all in the field of machine learning. Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the AI system that acquires sensor data, generates feature data, and determines a performance metric of a care provider taught by Jarc with displaying the text instructions to an augmented-reality device with input information taught by Qiu in order to provide real-time remote mentoring while using the inherent advantages of AR devices over traditional HUDs. ([0216] “In at least one embodiment, the AR system 100 allows real-time remote mentoring of the guided AR intervention. The remote mentoring can be accomplished by carrying out additional steps to those of method 4600. These additional steps are described as follows. (1) The replicate client device 170a receives real-time input data from the input device of the replicate client device 170a. The real-time input may help provide instruction and context from expert to novice.”)
Claim 14 is rejected under the same rationale as claim 4.
Response to Arguments
Claim Rejections - 35 USC § 101
The 35 USC 101 rejection has been withdrawn in view of the instant amendment.
Claim Rejections - 35 USC § 103
Applicant argues “While Jarc discloses neural networks in the abstract, Jarc does not disclose os suggest generating image features using a trained image feature recognition model nor decoding those image features using a separate trained procedure model to determine a step identifier.” Examiner respectfully disagrees. Jarc (Paragraphs [093], [094] and [113]) discloses a task assessment module that receives the image data uses a model to classify said data and then a surgeon assessment that uses the data classified under the task assessment to produce the particular step of the surgery.
Applicant also argues “Wolf therefore does not disclose or suggest determining a current step identifier during ongoing performance of a surgical procedure, using such a step identifier to retrieve step-specific instructions, or displaying such instructions to a care provider during performance of the procedure.” Examiner respectfully points to Jarc for teaching determining the current step of the surgery, Wolf is relied upon for retrieving and displaying the instructions of how to perform a step. The combination of references teaches presenting the instructions.
Applicant also argues “Neither Jarc nor Wolf disclose the use of sensors attached to a care provider” Examiner agrees and has introduced the King reference to teach a sensor attached to a care provider.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIELA D REYES whose telephone number is (571)270-1006. The examiner can normally be reached Monday-Friday, 7:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Wiley can be reached at (571) 272-3923. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142
Prosecution Insights