DETAILED ACTION
Introduction
Applicant's submission filed on 11/20/24 has been entered. Claims 1-20 are pending in the application and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Stoyanov et. al US PgPub. 2026/0164056 in view of Dhariwal et. al. US Patent 12,579,774.
Regarding claim 1, Stoyanov teaches a system for generating timestamped transcriptions using a local audio machine learning model data architecture to control compression parameters for converting local video recording data captured in an operating room or therapeutic facility prior to transmission to a remote machine learning controller configured to process the converted recording data using a trained machine learning model data architecture (Stoyanov, Fig. 1 and 2), the system comprising: a computer memory; non-transitory computer readable storage media; a processor configured to: record one or more raw audio data recordings in or proximate to the operating room or the therapeutic facility (Stoyanov [0032] A video may also come from a camera mounted in the operating room and external to the surgical site. The video that is captured can be transmitted and/or recorded in one or more example); conduct initial pre-processing against the one or more raw audio data recordings using the local audio machine learning model data architecture to transform the one or more raw audio data recordings into a set of intermediate pre-processing recording audio token data objects representative of time-stamped words or phrases (Stoyanov, [0075] At block 306, the machine learning execution system 240 uses the trained machine learning models 230 to detect one or more structures in the portion 406 of the video 404 being analyzed. Stoyanov, [0047] the machine learning models can also use audio data captured during the surgical procedure. The audio data can include sounds emitted by the surgical instrumentation system 106 while activating one or more surgical instruments 108. Alternatively, or in addition, the audio data (audio token objects) can include voice commands, snippets, or dialog from one or more actors 112. The audio data can further include sounds made by the surgical instruments 108 during their use ); using machine classification, cluster the set of intermediate pre-processing recording audio token data objects to establish timestamped durations of time, each duration of time corresponding to a data entry having a field type in a data structure(Stoyanov, [0036] For example, the machine learning models (e.g., the fully convolutional network adaptation) can be configured to perform supervised, self-supervised, or semi-supervised semantic segmentation(audio-visual semantic segmentation/classification) in multiple classes-each of which corresponding to a particular surgical instrument, anatomical body part (e.g., generally or in a particular state), and/or environment. [0038] After training, the one or more machine learning models can then be used in real-time to process one or more data streams (e.g., video streams, audio streams, RFID data, etc.). The processing can include predicting and characterizing one or more surgical phases, instruments, and/or other structures within various instantaneous or block time periods.); generate a media conversion instruction set data object including encoder parameter instructions for encoding the local video recording data, the encoder parameter instructions dynamically modified based on the field type corresponding to each duration of time for each data entry in the data structure, the encoder parameter instructions adapted to selectively maintain fidelity from the local video recording data(Stoyanov [0076-0077] At block 310, the computing system 102 selects a compression protocol to be applied to portion 406 of the video 404. The selected compression protocol can be different from the one used for other portions in the existing video 404(dynamically modified). Blur or enhance etc. Stoyanov [0082-0083] discusses selective compression protocols for different fidelity ); encode the local video recording data based at least on the media conversion instruction set data object to compress the local video recording data(see Stoyanov, [0071] The frames 408 in each video 404 can be encoded based on the format and/or codec used to store that video 404); and transmit the encoded recording data across a network to the remote machine learning controller, the remote machine learning controller configured to process the encoded recording data using the trained machine learning model data architecture(see Stoyanov, [0084] The output can include transmitting the video 404 to a remote location over a communication network, for example, for streaming, for storing, etc. Alternatively, or in addition, the output can include storing the video 404 in the compressed version ).
Stoyanov teaches using machine classification, cluster the set of intermediate pre-processing recording audio token data objects to establish timestamped durations of time, each duration of time corresponding to a data entry having a field type in a data structure, however to further compact prosecution and teach audio token data objects representative of time-stamped words or phrases, Dhariwal is used to further teach conduct initial pre-processing against the one or more raw audio data recordings using the local audio machine learning model data architecture to transform the one or more raw audio data recordings into a set of intermediate pre-processing recording audio token data objects representative of time-stamped words or phrases( see Dhariwal, Fig. 8 step 840 (1050 Speech ML model ; Dhariwal, col 14 lines 25-46 discuss keyword with respect to surgical procedures; Dhariwal col 18 lines 35-46 discusses tagging the video recording based on the keyword trigger ); using machine classification, cluster the set of intermediate pre-processing recording audio token data objects to establish timestamped durations of time, each duration of time corresponding to a data entry having a field type in a data structure (Dhariwal, Fig. 8, In step 840, when a keyword that triggers image or video capture is identified in the transcription, the images or video portion is recorded and stored. The images or video portion may be tagged with a timestamp or other metadata extracted from the transcription; Dhariwal, col 18 lines 56-58 discusses a Speech ML model to recognize keywords associated with macros and tasks (machine classification) ).
Stoyanov and Dhariwal are considered to be analogous to the claimed invention because both relate to processing of audio and video recordings during medical procedures. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Stoyanov to select, by the processor, a compression protocol for the portion of the video stream with the audio tagging of multimedia clips of the surgery of Dhariwal to improve improved automatic generation of operative reports (a report written in a patient's medical record to document the details of a surgery) including automatically extracting and tagging salient images or video clips of the surgical procedure (see Dhariwal, col 1 lines 55-67).
Regarding claim 2, Stoyanov in view of Dhariwal teaches system of claim 1. Dhariwal further teaches the local audio machine learning model data architecture is a trained domain specific audio machine learning model data architecture trained using data sets corresponding to the data structure (see Dhariwal, col 20 lines 59 – col 21 line 1, Fig. 10 ( training of Speech SL model, Dhariwal, col 14 lines 25-49 has the speech commands specific to surgery). The same motivation to combine as claim 1 applies here.
Regarding claim 3, Stoyanov in view of Dhariwal teaches system of claim 1. Dhariwal further teaches wherein the data structure includes data entries that correspond to specific steps of a surgical safety checklist (see Dhariwal, col 15 line 65- col 16 line 9 FIG. 6 shows an exemplar operative record 600 generated by the operative record generation system 200, according to some embodiments. The report 600 (operative record) is generated by collating all of the content, video, voice, and images in a specified format that is determined by the selected template in the system 200. The report generation process identifies “required” content and “optional” content, which is controlled by the surgeon and/or institution and set in the system preferences. The operative report 600 goes through a checklist of items, order of items, and format to generate the final report 600. Everything is timestamped and tagged to allow for validation of user and changes). Stoyanov further teaches wherein the data structure includes data entries that correspond to specific steps of a surgical safety checklist and each of the specific steps and the corresponding data entries include specific encoder parameter instructions adapted to require reduced compression of at least one of audio and video during the corresponding duration of time (see Stoyanov, [0064] discusses procedural tracking data structure list 255 (safety list), [0075-0076, 0078] discusses predicting the maneuver of surgical procedure to determine the compression protocol selection(specific encoder parameter instructions) for the portion 406 ). The same motivation to combine as claim 1 applies here.
Regarding claim 4, Stoyanov in view of Dhariwal teaches system of claim 3. Stoyanov further teaches wherein the required reduced compression is established through using variable compression mechanisms the during encoding of the local video recording data (see Stoyanov, [0076] At block 310, the computing system 102 selects a compression protocol to be applied to portion 406 of the video 404. The selected compression protocol can be different from the one used for other portions in the existing video 404(variable compression mechanisms). For example, different compression protocols can vary the resolution, framerate, pixelation, and any other image parameters that may be relevant to the video storage).
Regarding claim 5, Stoyanov in view of Dhariwal teaches system of claim 4. Stoyanov further teaches wherein the required reduced compression is established through using variable bit-rate compression the during encoding of the local video recording data (see Stoyanov [0076] The compression protocol can be any of the known protocols, such as H.264, HEVC, THEORA, AV1, VP9, RV40, or those developed in the future. (H.264, HEVC, THEORA, AV1, VP9, and RV40—supports or utilizes variable bit rate (VBR) compression)).
Regarding claim 6, Stoyanov in view of Dhariwal teaches system of claim 1. Dhariwal further teaches wherein the data structure is dynamically generated based on transcription tokens indicative of specific surgical instruments being used or procedures being conducted, and each of the data entries are dynamically associated with a period of time proximate to the timestamp of the transcription token indicative of specific surgical instruments being used or procedures being conducted (see Dhariwal, col 14 lines 25-29 In some embodiments, the keywords may be obtained from predefined templates associated with various surgical procedures. For example, an operative record template for an appendectomy may include keywords or key phrases that describe salient aspects of the surgical procedure(procedures being conducted or template user defined); Dhariwal, Claim 1 identify one or more predetermined keywords in the transcribed text from a predefined template associated with a type of surgical procedure being performed, wherein the keywords describe salient procedural aspects of the surgical procedure; input the one or more second features into the trained image extraction machine learning model to output second salient images from the surgical video stream in response to identifying the keywords; and tag the second salient images with a second identifier indicating keyword-triggered extraction). The same motivation to combine as claim 1 applies here.
Regarding claim 7, Stoyanov in view of Dhariwal teaches system of claim 6. Stoyanov further teaches wherein each of the specific surgical instruments being used or procedures being conducted are used for comparison against a reference data structure to establish then specific encoder parameter instructions adapted to require reduced compression of at least one of audio and video during the corresponding duration of time (see Stoyanov [0075-0076] At block 306, the machine learning execution system 240 uses the trained machine learning models 230 to detect one or more structures in the portion 406 of the video 404 being analyzed. Maneuver detector 250 predicts a maneuver of the surgical procedure captured in the selected portion(s) 404 based on the detected structures, At block 310, the computing system 102 selects a compression protocol to be applied to portion 406 of the video 404; Stoyanov, [0079, 0080, 0082] , based on the predetermined mapping, maneuvers, which are captured when camera 105 is outside patient 110, are removed/minimized (e.g., using a compression protocol that can cause pixelated/blurred portion). For the phases in which critical parts of the procedure are performed, e.g., critical view of safety (CVS), the information is stored with full fidelity, e.g., a lossless compression protocol. In some examples, actor 112 (e.g., User/surgeon/hospital) can manually specify the predetermined mapping of maneuvers and compression protocols. Further, actor 112 can also provide settings to use for a specific compression protocol. Also, certain parts of the maneuvers may be stored at reduced fidelity in either spatial or temporal resolution or in codec selection due to inactivity in the view).
Regarding claim 8, Stoyanov in view of Dhariwal teaches system of claim 1. Stoyanov further teaches wherein durations of time in the local video recording data that are not associated with a duration of time in the media conversion instruction set data object are not encoded in the encoded recording data (see Stoyanov, [0076] In some examples, portion 406 can be deleted or marked for deletion as part of the compression(not encoded)[0077] Partial-spatial compression protocols apply re-encoding filters or higher compression rates to a portion of a frame 408 (e.g., pixelate or blur out of focus areas, non-relevant organs, or regions of the frame 408)).
Regarding claim 9, Stoyanov in view of Dhariwal teaches system of claim 1. Stoyanov further teaches wherein the local audio machine learning model data architecture operates on local computing infrastructure of the operating room or therapeutic facility and is configured for separate computing operation from electronic recorders capturing the local video recording data (see Stoyanov [0049- 0050] A data collection system 150 can be employed to store the surgical data. The data collection system 150 includes one or more storage devices 152. The data collection system 150 can be a local storage system, a cloud-based storage system, or a combination thereof. [0056] The model execution system 240 can be separate from the machine learning training system 225 in some examples).
Regarding claim 10, Stoyanov in view of Dhariwal teaches system of claim 9. Stoyanov further teaches wherein the remote machine learning controller is electronically coupled across a plurality of network connections, each network connection coupled to an operating room or therapeutic facility of a plurality operating rooms or therapeutic facilities, the plurality of network connections having a limited amount of network bandwidth, and wherein the encoding based at least on the media conversion instruction set data object compresses the local video recording data before transmission to conserve network bandwidth across the plurality of network connections (see Stoyanov, [0088] Aspects of the technical solutions described herein can improve CAS systems, particularly by facilitating video storage optimization. Optimized/selective compression described herein improves (i.e., reduces) storage requirements. Aspects of the technical solutions described herein can also improve video transmission. The optimized/selective compression can be used to improve (i.e., reduce) network bandwidth requirements).
Regarding claim 11, is directed to a method claim corresponding to the system claim presented in claim 1 and is rejected under the same grounds stated above regarding claim 1.
Regarding claim 12, is directed to a method claim corresponding to the system claim presented in claim 2 and is rejected under the same grounds stated above regarding claim 2.
Regarding claim 13, is directed to a method claim corresponding to the system claim presented in claim 3 and is rejected under the same grounds stated above regarding claim 3.
Regarding claim 14, is directed to a method claim corresponding to the system claim presented in claim 4 and is rejected under the same grounds stated above regarding claim 4.
Regarding claim 15, is directed to a method claim corresponding to the system claim presented in claim 5 and is rejected under the same grounds stated above regarding claim 5.
Regarding claim 16, is directed to a method claim corresponding to the system claim presented in claim 6 and is rejected under the same grounds stated above regarding claim 6.
Regarding claim 17, is directed to a method claim corresponding to the system claim presented in claim 7 and is rejected under the same grounds stated above regarding claim 7.
Regarding claim 18, is directed to a method claim corresponding to the system claim presented in claim 8 and is rejected under the same grounds stated above regarding claim 8.
Regarding claim 19, is directed to a method claim corresponding to the system claim presented in claim 9 and is rejected under the same grounds stated above regarding claim 9.
Regarding claim 20, is directed to a non-transitory computer readable medium claim corresponding to the system claim presented in claim 1 and is rejected under the same grounds stated above regarding claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Grantcharov et. al. US Patent 12,11,4986 teaches “wherein the computer processor is further configured to request additional bandwidth resources for transmission of the captured video or audio data streams to the centralized computer server, the increased data volume or load adapted to support generation of one or more prediction data objects representative of one or more predicted characteristics or incidents relating to the medical procedure that occur during the one or more abnormality-related durations of time” (see Grantcharov, claim 1).
Pinto, US PgPub 2021/0098098 teaches Automated Clinical Documentation (ACD) process 10 may determine 404 a relative importance of a word in the report. ACD process 10 may determine 406 a portion of the video encounter information that corresponds to the word in the report. ACD process 10 may store 408, at a first location, the portion of the video encounter information that corresponds to the word in the report, wherein the video encounter information may be stored at a second location remote from the first location. (see Pinto, 0054).
Radhakrishnan et. al. US PgPub. 2007/0162924 teaches classification of segments of a video using an audio signal of the video and a set of classes based on different defined tasks (see Radhakrishnan, Fig. 3, Fig. 5).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANDINI SUBRAMANI whose telephone number is (571)272-3916. The examiner can normally be reached Monday - Friday 12:00pm - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NANDINI SUBRAMANI/ Examiner, Art Unit 2656
Prosecution Insights