DETAILED ACTION
This office action is responsive to communication(s) filed on 2/23/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims Status
Claims 1-2, 6-7, and 10-25 are pending and are currently being examined.
Claims 1 and 19-20 are independent.
Claims 3-5 and 8-9 are newly canceled.
Claims 21-25 newly added.
Claims 1-2, 10, 12-15 and 17-20 are newly amended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-7, 10-11, 13-14, and 16-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf; Tamir et al. (hereinafter Wolf – US 20200237452 A1) in view of Sachdeva; Rohit C. et al. (hereinafter Sachdeva – US 20200066391 A1).
Independent Claim 1:
Wolf teaches:
A computer-implemented method, comprising:
obtaining one or more images of a surgical procedure; (accessing video footage at 802, e.g., ¶ 188 and figs. 4-5 and 8A. capture of video and audio data, as well as data from other sensors, during the surgical procedure, ¶ 85)
determining, using one or more machine learning models, a set of images from the one or more images based on the surgical procedure; (a machine learning model uses image recognition algorithms to analyze video frames/images and generate markers for frame(s), by associating the frame(s) with certain surgical procedures, surgical phases, intraoperative events, and/or event characteristics, together with labels indicating locations within the video footage, ¶¶ 114 and 116 and figs. 4-5)
and generating an interactive video surgical report for the surgical procedure comprising at least some of the set of images (see Abstract and ¶ 110 and figs. 4-5. Overlaying a surgical timeline with markers onto a surgical video transforms raw video data into a structured, searchable, and clinically meaningful record, reflecting the generation of a modern, dynamic, and interactive video surgical report, at least one in its preliminary stage. The report is “interactive” at least because it is searchable and dynamic, based on user interaction with one or more of the markers, Abstract)
[…].
Wolf does not appear to expressly teach, but Sachdeva teaches:
and a virtual character configured to output a response to a question about the surgical procedure received from a patient via a user interface associated with display of the interactive video surgical report. (The VCN [virtual care navigator] bot uses an animated 3D avatar as its interface to provide context-sensitive, on-demand guidance and answers to patient queries through various input/output modalities, ¶¶ 201 and 223. Additionally, the system is designed to process patient questions and deliver responses via the bot's interface, allowing the avatar to act as a medium for these interactions, ¶ 202. FIG. 15 illustrates a virtual care navigator (VCN) system, see fig. 15 and ¶ 89. The patient-facing workflow, outlined in steps 1708-1710, enables users to receive and inquire about treatment progress, doctor reports, and consultation updates through an AI-driven platform, ¶¶ 201,202,223 and fig. 17. The treatment is a surgery, see ¶¶ 142-143, and a treatment plan is displayed for the patient, ¶ 120, and the system can be used for the entire workflow, from planning to post-active treatment, ¶ 12. Here, the treatment is a surgery because the software allows doctors to plan and execute specific “surgical movements” on a patient’s “bony structure” and “bone soft tissue” [about the surgical procedure], ¶¶ 142-143. The patient is the patient for the treatment [surgery], and the VCN, when guiding the patient, determines the user’s status, e.g., close to the end of their treatment plan, as such the patient questions are also interpreted as being “about the surgical procedure”. It was well within the capabilities of a person having ordinary skill in the art to have realized that in implementing the concepts of Sachdeva in method of Wolf would allow for interactive management of the personal care of the user at any stage, e.g., including during/after outputting of the video surgical report of Wolf).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Wolf to include and a virtual character configured to output a response to a question about the surgical procedure received from a patient via a user interface associated with display of the interactive video surgical report, as taught by Sachdeva.
One would have been motivated to make such a combination in order to improve the functionalities and user experience by empowering the patient to manage their personal care when appropriate, Sachdeva ¶ 7.
Claim 2:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein generating the interactive video surgical report comprises:
generating, using the one or more machine learning models, text describing the at least some of the set of images, wherein the interactive video surgical report comprises at least some of the text corresponding to the at least some of the set of images. (the markers may include text-based titles or descriptions referring to a particular location of the video, ¶ 159.)
Claim 6:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein the one or more images include a first image and a second image captured during a same phase of the surgical procedure, (as mentioned above, the machine learning model uses image recognition algorithms to analyze video frames/images and generate markers for frame(s), by associating the frame(s) with certain surgical procedures, surgical phases, intraoperative events, and/or event characteristics, together with labels indicating locations within the video footage, ¶¶ 114 and 116 and figs. 4-5. The intraoperative event identified from the different frames/video footage may be surgical events within surgical phases, ¶¶ 147 and 189, and the location of the markers may be particular frames or range of frames [include a first image and a second image] in the video, ¶ 112)
determining the set of images comprises:
computing, using the one or more machine learning models, a first classification score and a second classification score respectively associated with the first image and the second image; (wherein the one or more estimated outcomes are a result of an analysis of a plurality of videos of past surgical procedures including respective similar decision making junctions, ¶ 764, and respective similar decision making junctions are similar to the decision making junction of the surgical procedure according to a similarity metric, ¶ 774, the similarity measure/metric [a first classification score and a second classification score] may be selected if they are above a threshold, ¶¶ 129 and 230)
and adding at least one of the first image or the second image to the set of images based on the first classification score and the second classification score. (the tags group are added to all frames identified to specific events within the phase, ¶ 230)
Claim 7:
The rejection of claim 1 is incorporated. Wolf further teaches:
further comprising:
identifying at least one image from the one or more images based on preoperative information related to the surgical procedure, wherein the set of images comprises the at least one image. (the activities/events detected may include preoperative activity [information], ¶ 196. the information that distinguishes portions of the historical surgical footage into frames associated with an intraoperative surgical event may include detected tools and anatomical features in associated frames. For example, the disclosed methods may include using an image and/or video analysis algorithm to detect tools and anatomical features. The tools may include surgical tools, as described above, or other nonsurgical tools, ¶ 200. By definition, non-surgical tool is associated with nonsurgical, preoperative or postoperative activities)
Claim 10:
The rejection of claim 1 is incorporated. Wolf further teaches:
further comprising:
receiving preoperative information related to at least one of the surgical procedure or a patient associated with the surgical procedure; (information received and detected in the video frames may be identified as preoperative activity/events, ¶ 196. )
and generating content to be included in the interactive video surgical report based on the received preoperative information. (analyze video frames/images and generate markers for frame(s), by associating the frame(s) with certain surgical procedures, surgical phases, intraoperative events [activity], and/or event characteristics, together with labels indicating locations within the video footage, ¶¶ 114 and 116 and figs. 4-5)
Claim 11:
The rejection of claim 1 is incorporated. Wolf further teaches:
further comprising:
generating, using the one or more machine learning models, text associated with the set of images; (the markers may include text-based titles or descriptions referring to a particular location of the video, ¶ 159. The location of the markers may be particular frames or range of frames [set of images] in the video, ¶ 112)
and associating one or more portions of the text with the set of images. (the markers may include text-based titles or descriptions referring to a particular location of the video, ¶ 159.)
Claim 13:
The rejection of claim 1 is incorporated. Wolf further teaches:
further comprising:
receiving, using one or more audio sensors, audio captured during the surgical procedure, (facilitating the capture of video and audio data, as well as data from other sensors, during the surgical procedure, ¶ 85)
wherein the interactive video surgical report comprises at least some of the audio corresponding to the at least some of the set of images. (the annotated, time-lined video includes audio captured during the surgical procedure, ¶ 442)
Claim 14:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein generating the interactive video surgical report comprises:
obtaining pre-generated text associated with content depicted by the at least some of the set of images; (the markers may include text-based titles or descriptions [pre-generated text] referring to a particular location of the video, ¶ 159)
obtaining user-provided text of audio captured during the surgical procedure; (an event, in which the abovementioned identification of event and tag/label, may be a surgeon’s spoken word during a surgery, ¶ 361)
and generating text for the interactive video surgical report based on the pre-generated text and the user-provided text. (Overlaying, based on spoken words and other identified features, a surgical timeline with markers onto a surgical video, Abstract and figs. 4-5 and ¶ 361)
Claim 16:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein obtaining the one or more images comprises:
accessing video captured during the surgical procedure; (accessing video footage at 802, e.g., ¶ 188 and figs. 4-5 and 8A. capture of video and audio data, as well as data from other sensors, during the surgical procedure, ¶ 85)
and extracting at least one video snippet from the video based on the surgical procedure, wherein the one or more images comprise the at least one video snippet. (the detection and marker creation location can be a particular frame or range of frames in the video [video snippet], a particular timestamp, or any other indicator of position within the video, ¶ 112)
Claim 17:
The rejection of claim 1 is incorporated. Wolf further teaches:
further comprising:
obtaining one or more additional images captured subsequent to the surgical procedure; (video/image data needs to be captured during, before, or after a surgical procedure, ¶ 302)
Wolf further suggests:
and generating an updated interactive video surgical report comprising at least some of the one or more additional images. (Wolf teaches an embodiment in which a post-operative report may be populated [generating] with information from a surgical footage, ¶ 415. Wolf also teaches that while illustrative embodiments have been described, the scope may include any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations or alterations based on the present disclosure, ¶ 1161).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Wolf to include and generating an updated interactive video surgical report comprising at least some of the one or more additional images, as suggested by Wolf.
One would have been motivated to make such a combination in order to adjust the scope of Wolf by combining the embodiment of preliminary video report reflected in the mapping of claim 1 with the embodiment of populating a post-operative report, Wolf ¶¶ 415 and 1161. It was well within the capabilities of a person having ordinary skill in the art to have realized that in the preliminary report in the first embodiment would could be enhanced and/or finalized into an official post-operative report.
Claim 18:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein generating the interactive video surgical report comprises:
adding one or more additional images to the interactive video surgical report based on a similarity between content depicted by the one or more additional images and content depicted by at least one of the set of images, wherein the one or more additional images are captured prior to the surgical procedure. (the particular surgical footage may contain a second group of frames not associated with surgical activity, ¶ 196)
Independent Claims 19-20:
Claim(s) 19-20 is/are directed to a computer-readable medium and system for accomplishing the steps of the method in claim 1, and are rejected using similar rationale(s).
Claim 21:
The rejection of claim 1 is incorporated. Sachdeva further teaches:
wherein the virtual character represents at least one aspect of a surgeon that performed at least a portion of the surgical procedure. (user of the system include doctors, ¶ 130, and the avatars may have the likeness [aspect] of the user, e.g., a doctor avatar, ¶¶ 157 and 197)
Claim 22:
The rejection of claim 21 is incorporated. Sachdeva further teaches:
wherein the at least one aspect comprises a voice of the surgeon or an appearance of the surgeon. (user of the system include doctors, ¶ 130, and the avatars may have the likeness [appearance] of the user, e.g., a doctor avatar, ¶¶ 157 and 197)
Claim 23:
The rejection of claim 1 is incorporated. Sachdeva further teaches:
wherein the virtual character is configured to describe at least one surgical event based on the one or more images. (The virtual character (VCN bot) is configured to describe a surgical event by guiding the patient through a treatment roadmap step-by-step that uses “image/video simulations” to demonstrate how specific “dental malocclusions will be fixed”, ¶ 203. These steps, or sequenced events, are e.g., alignment, space closure, root correction, jaw repositioning and orthopedics and stabilization, finishing, and retention, ¶ 257. The avatar is connected to the bot, ¶ 197, and the VCN bot and all its interactions with the user take place through the user's realistic personalized 3D avatar, unless user disables this function, ¶ 199)
Claim 24:
The rejection of claim 23is incorporated. Sachdeva further teaches:
wherein the at least one surgical event is captured in at least one image displayed concurrently with the virtual character describing the at least one surgical event. (the system enables a personalized 3D avatar to concurrently display image simulations of specific dental procedures while narrating the treatment steps, allowing the patient to visualize how their malocclusion will be corrected, ¶¶ 203 and 257)
Claim 25:
The rejection of claim 23is incorporated. Sachdeva further teaches:
wherein the at least one surgical event is depicted in an animation displayed concurrently with the virtual character describing the at least one surgical event. (the avatar guides patients through a treatment roadmap by displaying step-by-step, animated simulations of dental corrections, which are synchronized with the personalized 3D avatar’s description of each event, ¶¶ 203 and 257)
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf (US 20200237452 A1) in view of Sachdeva (US 20200066391 A1), as applied to claim 1 above, and further in view of Avijeet; Vijeta (hereinafter Avijeet – US 20220130378 A1).
Claim 12:
The rejection of claim 1 is incorporated. Wolf further teaches:
wherein generating the interactive video surgical report comprises:
generating, using the one or more machine learning models, text for the at least some of the set of images; (the markers may include text-based titles or descriptions referring to a particular location of the video, ¶ 159).
Wolf does not appear to expressly teach, but Avijeet teaches:
and generating, using the one or more machine learning models, audio based on the text, the interactive video surgical report comprising the audio (a natural language generator (NLG) neural network that may generate narratives and reports in easy-to-read language, and a text-to-speech (TTS) neural network that may convert text to speech imitating a speaker's voice, ¶ 35).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Wolf to include and generating, using the one or more machine learning models, audio based on the text, the interactive video surgical report comprising the audio, as taught by Avijeet.
One would have been motivated to make such a combination in order to improve the functionality and usability of the method by providing an accurate, easy to understand, dialogue-based interaction with the user, Avijeet Abstract and ¶¶ 3 and 34-35.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf (US 20200237452 A1) in view of Sachdeva (US 20200066391 A1), as applied to claim 1 above, and further in view of Richter; Ian M. et al. (hereinafter Richter – US 12249253 B1).
Claim 15:
The rejection of claim 1 is incorporated. Wolf does not appear to expressly teach, but Richter teaches:
wherein generating the interactive video surgical report comprises:
generating audio based on data stored in an audio profile of a user, the data comprising at least one of a pitch, a timbre, a loudness, or a modulation associated with the user. (a response synthesizer modulates the tonal qualities, amplitude, inflection, accent, rate of speech, etc., of the audio content item based on one or more user characteristics relating to the user. For example, the response generator may develop a user profile corresponding to the user over time, e.g., based on usage history, and modulate aspects of the audio content item based on the user's preferences or based on historical data relating to the user's level of engagement with the conversational agent, col 16:47-56. This reflects “modulation associated with the user”, and modulating qualities like amplitude, inflection, accent, and rate of speech relates to all three concepts—pitch, timbre, and loudness—as they are interconnected parts of voice modulation and prosody.)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Wolf to include wherein generating the interactive video surgical report comprises: generating audio based on data stored in an audio profile of a user, the data comprising at least one of a pitch, a timbre, a loudness, or a modulation associated with the user, as taught by Richter.
One would have been motivated to make such a combination in order to improve usability of the method by increasing using engagement with the content, Richter col 16:27-46.
Response to Arguments
Applicant’s 102/103 arguments, Remarks Pg(s) 7-9, have been considered but are moot in view of the new grounds of rejection presented above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Below is a list of these references, including why they are pertinent:
Casey; Niall Patrick et al. US 20240203551 A1, is pertinent to claim 1 for disclosing that FIG. 8 illustrates an interactive, patient-facing surgical plan report that displays customized medical data and allows users to view 3D models, annotate details, and securely submit questions back to their healthcare providers, ¶ 123.
Poltaretskyi; Sergii et al. US 20190380792 A1, is pertinent to claim 1 for disclosing, in an orthopedic surgical system, Mixed Reality (MR) visualization devices allow patients to view interactive, virtual 3D models of their condition and the upcoming procedure during the preoperative planning phase, ¶ 218.
Morgan; Joseph W. et al. (hereinafter Morgan – US 10943407 B1), is pertinent to claim 1, for disclosing that audio may be reflected as emanating from an avatar for provides audio instructions to a user, cols 38:29-46 and 64:6-18, and the instructional avatar can be presented together with text-based, image-based and/or video-based instructional content, col 32:56-62.
Jonelagadda; Chandra et al. US 20250307887 A1, is pertinent to claim 1 for disclosing systems and methods for generating a surgery report based on a video of a surgical procedure (“surgical video”). The surgical video may be analyzed using one or more neural networks (e.g., artificial intelligence) that have been trained to identify patient anatomy, surgical tools, and/or patient pathologies, ¶ 34.
Jain; Abhinav et al. US 11935170 B1, is pertinent to claim 3 for disclosing a digital avatar presented as a graphical overlay during playback of the video content, col 7:4-19.
Horiuchi; Makoto US 20190304591 A1, is pertinent to claim 12 for disclosing the conversion into voice is processing in which the character strings (text data) are converted into voice data, and a known software of text-to-speech may be used for the processing, ¶ 122
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL S MERCADO whose telephone number is (408)918-7537. The examiner can normally be reached Mon-Fri 8am-5pm (Eastern Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gabriel Mercado/Primary Examiner, Art Unit 2171
Prosecution Insights