DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a) because the drawings fail to show label 515 and 516 in paragraph 58, 60, 77, 78 and fail to show figure 6 as indicated in paragraph 77 as described in the specification. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The abstract of the disclosure does not commence on a separate sheet in accordance with 37 CFR 1.52(b)(4) and 1.72(b). A new abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text.
The disclosure is objected to because of the following informalities:
Figure 3 discloses the label 212 however, the specifications do not. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 10-12, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piskun et al. (U.S. Pub. No. 20220093236) in view of Meagher et al. (U.S. Pub. No. 20200297433).
Regarding claim 1, Piskun discloses a plurality of user interface elements, each user interface element respectively corresponding to a structure (para 48, “For example, as show in FIG. 7, an exemplary image of the video overlay can be provided indicating four safe to proceed markers 705, 710, 715, and 720.”; also, para 48, “For example, marker 705 can represent a vein and marker 710 can represent a nerve.”; also, para 48, “Markers 705 and 710 can be associated with anatomical structures 805 and 810, respectively.”; the video overlay contains markers that is a form of a user interface elements while each marker being associated with an anatomical structure) from a list of structures anticipated in a surgical video (para 44, “For example, prior to the surgical procedure being performed, the exemplary system, method, and computer-accessible medium can be provided with information on the surgical procedure (e.g., the type, what is being performed, etc.).”; also, para 44, “The programmed computer processor(s) used with and/or by the exemplary system, method, and computer-accessible medium can access a database that includes markers associated with the surgical procedure.”; Since when a surgical procedure can be provided with information prior to a surgery, a list of anatomical structures can also be indicated and pulled from the database), each user interface element having a visual attribute (para 48, “As shown in FIG. 7, markers 705-720 are all one color (e.g., red), indicating that the exemplary system, method, and computer-accessible medium has not located the markers.”; visual distinction between placed user interface elements and markers that have not been placed), wherein the visual attribute of a first user interface element is set to a first state in response to a first structure being detected in the surgical video in a (para 48, “The color associated with markers 705 and 710 can change (e.g., from red to green) to indicate to the surgeon that these markers have been identified.”; also, para 43, “In particular, the programmed computer processor(s) used with and/or by the exemplary system, method, and computer-accessible medium can analyze the live video from a surgical procedure to determine the presence or absence of particular markers needed to proceed during the surgery.”; When such a structure has been identified, the visual attribute will change to indicate the discovery of such structure based on based on the live surgical video and since when each element is being displayed in the overlay of the video, that would effectively be the field of view of the surgeon), and the visual attribute of the first user interface element is set to a second state in response to the first structure not being detected in the (para 48, “As the surgeon/medical professional proceeds with the surgery/medical procedure, it can be possible that a marker previously identified by the exemplary system, method, and computer-accessible medium can no longer be identified”; also, para 48, “In such a case, the marker can revert back to the original color (e.g., red) indicating that the exemplary programmed computer processor(s) can no longer identify the specific marker.”; When a structure is no longer visible or can no longer be determined, such markers would change to a different visual attribute). Piskun does not disclose a computer-implemented user interface and a field of view.
However, in a similar field of endeavor, Meagher discloses a computer-implemented user interface (para 16, “Some implementations allow the surgeon to recall previously tagged structures or locations via the robotic user interface, and overlay a graphical tag over the selected location in the endoscopic view to facilitate navigation of the surgical space under compromised visual conditions.”; also, para 15, “A system in accordance with the present invention includes one or more information sources, such as 2D, 3D and/or structured light imaging sources, and a visual display for displaying visual information from those sources to a user.”; Interface for displaying a series of structures) and a field of view (para 16, “At a high level, this application describes a system and method for visually tracking locations or structures within a surgical endoscopic visual field of view as that view changes.”; also, para 27, “The system may thus be able to detect changes in the visibility of bookmarked features on the endoscopic view (or increases in the degree to which the bookmarked features are obscured).”; also, para 26, “Visual characteristics of the tags may be altered as a means of visual feedback to the surgeon relating to some changed condition, such as a change in visibility in the endoscopic field that compromises the visibility of identified structures on the endoscopic display.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention of a plurality of markers displayed on live surgical video, each marker corresponding to a structure drawn from a procedure-specific set and each carrying a color that is set to one value when the processor identifies the corresponding structure and reverts to another value when it can no longer identify it, with the features of Meagher's invention of conditioning the visual characteristics of a per-structure tag on that structure's visibility within the endoscopic field of view as that view changes. The combination would have been obvious because a person of ordinary skill working on Piskun's display would have wanted a marker that reports whether its structure lies within the view presently before the surgeon, and Meagher supplies that condition for a tag on the same kind of endoscopic feed. Each reference performs in the combination the function it performs separately, since Piskun's detection and marker rendering are unchanged and Meagher's visibility condition governs only which of the two states a marker takes, so the result would have been predictable to one of ordinary skill.
Regarding claim 2, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, wherein Piskun further discloses the visual attribute of the first user interface element is based on the first structure being marked with an overlay using said visual attribute (para 22, “As shown in FIG. 1, one or more computer processors used with and/or by the exemplary system, method, and computer-accessible medium can be programmed to generate a video overlay which can highlight (e.g., using any color), and/or otherwise identify various anatomical structures in the video.”; also, para 21, “The markings can be transparent highlights, which can be, for example, superimposed on the anatomy of interest (e.g., transparent punctate lines for predicting the anatomy and transparent solid lines for the identified anatomy) or portions thereof, while not interfering with the surgical video.”; also, para 48, “The color associated with markers 705 and 710 can change (e.g., from red to green) to indicate to the surgeon that these markers have been identified.”).
Regarding claim 6, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1 wherein Piskun further discloses the surgical video is a live video stream (para 22, “For example, one or more processors used with and/or by the exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can receive a live video of a medical/surgical procedure (e.g., a laparoscopic surgical procedure), and in real time, provide the surgeon or another medical professional with an overlay over the video.”; also, para 27, “The video overlay can be continuously modified, in real time, using the machine learning procedure.”).
Regarding claim 10, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1,wherein Piskun further discloses the visual attribute is one of a color, a pattern, a shape, an image, and an animation (para 48, “As shown in FIG. 7, markers 705-720 are all one color (e.g., red), indicating that the exemplary system, method, and computer-accessible medium has not located the markers.”; also, para 49, “The color associated with markers 715 and 720 has been changed (e.g., from red to green), indicating that these markers have been identified”;).
Regarding claim 11, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, wherein Piskun further discloses the user interface element comprises a label (para 48, “Each marker 705-720 can have an associated description, which can provide information to the surgeon on what the specific marker refers to.”; also, para 48, “For example, marker 705 can represent a vein and marker 710 can represent a nerve.”).
Regarding claim 12, Piskun discloses a computer-implemented method comprising (para 22, “For example, one or more processors used with and/or by the exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can receive a live video of a medical/surgical procedure (e.g., a laparoscopic surgical procedure), and in real time, provide the surgeon or another medical professional with an overlay over the video.”; also, para 55, “For example, exemplary procedures used by the exemplary system, method, and computer-accessible medium in accordance with the present disclosure described herein can be performed by a processing arrangement and/or a computing arrangement (e.g., computer hardware arrangement) 1105.”): identifying, by one or more processors, a structure in a video of a surgical procedure using machine learning (para 30, “For example, the exemplary machine learning procedure can be used to predict (i) the cystic artery with a mask, (ii) the cystic duct with a mask, and/or (iii) the location of the gallbladder, the target zone, and the danger zone with masks.”; also, para 23, “Gallbladder 105 has been highlighted by one or more programmed computer processors used with and/or by the exemplary system, method, and computer-accessible medium, using, for example, a machine learning procedure as discussed in further detail below.”; also, para 27, “The video overlay can be continuously modified, in real time, using the machine learning procedure.”); one or more processors, a user interface element corresponding to the structure using a first visual attribute (para 48, “The color associated with markers 705 and 710 can change (e.g., from red to green) to indicate to the surgeon that these markers have been identified.”; also, para 48, “Markers 705 and 710 can be associated with anatomical structures 805 and 810, respectively.”); one or more processors, the user interface element corresponding to the structure using a second visual attribute (para 48, “In such a case, the marker can revert back to the original color (e.g., red) indicating that the exemplary programmed computer processor(s) can no longer identify the specific marker.”). Piskun does not disclose in response to at least a portion of the structure being visible in a field of view and in response to the structure not being visible in the field of view.
However, in a similar field of endeavor, Maegher disclose in response to at least a portion of the structure being visible in a field of view (para 16, “At a high level, this application describes a system and method for visually tracking locations or structures within a surgical endoscopic visual field of view as that view changes.”) and in response to the structure not being visible in the field of view (para 26, “Visual characteristics of the tags may be altered as a means of visual feedback to the surgeon relating to some changed condition, such as a change in visibility in the endoscopic field that compromises the visibility of identified structures on the endoscopic display.”; also, para 27, “The system may thus be able to detect changes in the visibility of bookmarked features on the endoscopic view (or increases in the degree to which the bookmarked features are obscured).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention of a computer-implemented method in which one or more processors identify a structure in live surgical video with a convolutional neural network and represent that structure's marker in one color while the structure is identified and in another color once it can no longer be identified, with the features of Meagher's invention of visually tracking structures within the endoscopic field of view as that view changes and altering a tag when a structure's visibility in that field is compromised. The combination would have been obvious because a surgeon watching a moving camera needs to know whether a structure is presently in view, and Meagher conditions its tag on exactly that, on the same kind of endoscopic feed Piskun processes. Each reference performs in the combination the function it performs separately, since the visibility condition governs only which representation is selected and requires no change to Piskun's neural network, so the result would have been predictable to one of ordinary skill.
Regarding claim 18, Piskun as modified by Maegher discloses the computer-implemented method of claims 12, wherein Piskun further discloses graphical overlay with the same first visual attribute to highlight the structure (para 22, “As shown in FIG. 1, one or more computer processors used with and/or by the exemplary system, method, and computer-accessible medium can be programmed to generate a video overlay which can highlight (e.g., using any color), and/or otherwise identify various anatomical structures in the video.”; also, para 21, “The markings can be transparent highlights, which can be, for example, superimposed on the anatomy of interest (e.g., transparent punctate lines for predicting the anatomy and transparent solid lines for the identified anatomy) or portions thereof, while not interfering with the surgical video.”). Piskun does not disclose the first visual attribute is used to represent the structure in response to the structure being in the field of view.
However, in a similar field of endeavor, Maegher discloses the first visual attribute is used to represent the structure in response to the structure being in the field of view (para 16, “At a high level, this application describes a system and method for visually tracking locations or structures within a surgical endoscopic visual field of view as that view changes.”; also, para 26, “Visual characteristics of the tags may be altered as a means of visual feedback to the surgeon relating to some changed condition, such as a change in visibility in the endoscopic field that compromises the visibility of identified structures on the endoscopic display.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention of a video overlay that highlights an identified anatomical structure in color at its location in the surgical video, with the features of Meagher's invention of visually tracking structures within the endoscopic field of view as that view changes and altering the representation when a structure's visibility in that field is compromised. The combination would have been obvious because Piskun highlights whatever structure its model has identified without stating what governs whether the structure is there to be highlighted, and Meagher supplies the field-of-view condition that answers it. Each reference performs in the combination the function it performs separately, so the result would have been predictable to one of ordinary skill.
Regarding claim 19, Piskun as modified by Maegher discloses the computer-implemented method of claims 12, wherein Piskun further discloses the video is a live video stream of the surgical procedure (para 43, “In particular, the programmed computer processor(s) used with and/or by the exemplary system, method, and computer-accessible medium can analyze the live video from a surgical procedure to determine the presence or absence of particular markers needed to proceed during the surgery.”; also, para 22, “For example, one or more processors used with and/or by the exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can receive a live video of a medical/surgical procedure (e.g., a laparoscopic surgical procedure), and in real time, provide the surgeon or another medical professional with an overlay over the video.”).
Regarding claim 20, Piskun discloses a computer program product comprising a memory device having computer executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method (para 56, “As shown in FIG. 11, for example a computer-accessible medium 1115 (e.g., as described herein above, a storage device such as a hard disk, floppy disk, memory stick, CD-ROM, RAM, ROM, etc., or a collection thereof) can be provided (e.g., in communication with the processing arrangement 1105).”; also, para 56, “The computer-accessible medium 1115 can contain executable instructions 1120 thereon.”; also, para 56, “In addition or alternatively, a storage arrangement 1125 can be provided separately from the computer-accessible medium 1115, which can provide the instructions to the processing arrangement 1105 so as to configure the processing arrangement to execute certain exemplary procedures, processes, and methods, as described herein above, for example.”) method comprising: identifying, using a neural network model, a structure in a video of a surgical procedure, the neural network model is trained using surgical training data (para 30, “The exemplary machine learning procedure can be and/or include deep Convolutional Neural Network ("CNN") for Semantic Segmentation tasks.”; also, para 30, “For example, the exemplary machine learning procedure can be used to predict (i) the cystic artery with a mask, (ii) the cystic duct with a mask, and/or (iii) the location of the gallbladder, the target zone, and the danger zone with masks.”; also, para 23, “Gallbladder 105 has been highlighted by one or more programmed computer processors used with and/or by the exemplary system, method, and computer-accessible medium, using, for example, a machine learning procedure as discussed in further detail below.”; also, para 22, “For example, one or more processors used with and/or by the exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can receive a live video of a medical/surgical procedure (e.g., a laparoscopic surgical procedure), and in real time, provide the surgeon or another medical professional with an overlay over the video.”; also, para 30, “For the semantic segmentation model training and testing, the following exemplary datasets were used: (i) datasets containing frames from laparoscopic cholecystectomy videos, and (ii) datasets containing corresponding masks marked by surgeons.”; also, para 31, “For the cystic duct mask and the cystic artery mask, the exemplary training included 33635 frames from laparoscopic cholecystectomy videos and validation/testing included 15290 frames from laparoscopic cholecystectomy videos.”); generating a visualization that comprises a graphical overlay at a location of the structure in the video of the surgical procedure, the graphical overlay uses a first visual attribute (para 22, “As shown in FIG. 1, one or more computer processors used with and/or by the exemplary system, method, and computer-accessible medium can be programmed to generate a video overlay which can highlight (e.g., using any color), and/or otherwise identify various anatomical structures in the video.”; also, para 21, “The markings can be transparent highlights, which can be, for example, superimposed on the anatomy of interest (e.g., transparent punctate lines for predicting the anatomy and transparent solid lines for the identified anatomy) or portions thereof, while not interfering with the surgical video.”); identifying a symbol corresponding to the structure from a list of displayed symbols (para 48, “Markers 705 and 710 can be associated with anatomical structures 805 and 810, respectively.”; also, para 48, “For example, marker 705 can represent a vein and marker 710 can represent a nerve.”; also, para 48, “For example, as show in FIG. 7, an exemplary image of the video overlay can be provided indicating four safe to proceed markers 705, 710, 715, and 720.”; also, para 52, “As shown in FIGS. 7-9, markers 705-720 can be provided as part of the video overlay.”; also, para 52, “Alternatively or in addition, markers 705-720 can be provided on a separate video screen so as to not interfere with the surgeon's view of the procedure.”; also, para 47, “Based on this indication from the surgeon/medical professional, the exemplary programmed computer processor(s) can display all markers identified, but then mark certain ones as necessary and certain ones as beneficial (e.g., as indicated by the surgeon/medical professional) or only display the necessary markers.”; also, para 49, “For example, markers 715 and 720, which can correspond to anatomical structures 905 and 910, respectively, have then been identified by the exemplary programmed computer processor(s) used by or with the exemplary system, method, and computer-accessible medium.”; a plurality of symbols displayed as a group, from which the one corresponding to a given structure is picked out); and updating the symbol by displaying the symbol using the first visual attribute (para 48, “As shown in FIGS. 7-9, markers 705-720 can be provided as part of the video overlay.”; also, para 48, “FIG. 8 shows an exemplary image in which exemplary programmed computer processor(s) used by or with the exemplary system, method, and computer-accessible medium has identified two markers (e.g., markers 705 and 710).”; also, para 48, “color associated with markers 705 and 710 can change (e.g., from red to green) to indicate to the surgeon that these markers have been identified.”). Piskun does not disclose generating a user interface to depict presence of structures in a field of view.
However, in a similar field of view, Maegher discloses generating a user interface to depict presence of structures in a field of view (para 16, “At a high level, this application describes a system and method for visually tracking locations or structures within a surgical endoscopic visual field of view as that view changes.”; also, para 15, “A system in accordance with the present invention includes one or more information sources, such as 2D, 3D and/or structured light imaging sources, and a visual display for displaying visual information from those sources to a user.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention of a computer-accessible medium carrying executable instructions that, when executed by one or more processors, identify a structure in a video of a surgical procedure using a convolutional neural network trained on frames from laparoscopic videos and on masks marked by surgeons, generate a video overlay that highlights that structure in color at the structure's location in the video, display a set of markers each associated with a respective anatomical structure and identify from that displayed set the marker corresponding to the structure found, and update that marker, which is itself provided as part of the same video overlay, once the structure has been identified, with the features of Meagher's invention of a surgical display that visually tracks structures within the endoscopic field of view as that view changes. The combination would have been obvious because Piskun reports identification without stating what governs whether a structure is available to be identified, and Meagher supplies the field-of-view framing in which the presence of the structures is reported to the surgeon. Each reference performs in the combination the function it performs separately, since Meagher's framing leaves Piskun's detection, overlay and markers unchanged, so the result would have been predictable to one of ordinary skill.
Claim(s) 3, 8-9, 13-14, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piskun et al. (U.S. Pub. No. 20220093236) as modified by Meagher et al. (U.S. Pub. No. 20200297433), further in view of Barral et al. (U.S. Pub. No. 20180065248).
Regarding claim 3, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, first structure is detected in the field of view using machine learning.
However, in a similar field of endeavor, Barral discloses wherein the first structure is detected in the field of view using machine learning (para 40, “As another example, the classifiers analyze visual sensor data, such as endoscope data, to recognize anatomical structures in a field of view to determine if the expected anatomical structures are in view, recognize which medical tools are currently being used, whether a deviation in the medical procedure has occurred due to the view of unexpected anatomical structures, etc.”; also, para 24, “In one embodiment, the medical procedure model includes multiple classifiers, such as a step identification classifier that is used to identify a current step in a medical procedure (e.g., current step is X, next step is Y with potential deviation of Z), an anatomy classifier that is used to identify one or more anatomical structures during a medical procedure based on input video or still image data (e.g., identification of a patient's arteries, organs, etc.), and one or more tool classifiers that identify proper and improper usage of a tool during a medical procedure based on medical tool sensor data (e.g., identification of an expected incision depth, tool movement(s) direction, speed, acceleration, etc.).”; also, para 28, "As another example, machine learning training engine 212 may analyze the corpus of training set data in data store 214 utilizing deep neural networks to generate the step identification, anatomy, and tool classifiers.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention with the features of Barral's invention of classifiers, generated by a machine learning training engine using deep neural networks, that analyze endoscope data to recognize anatomical structures in a field of view and to determine whether the expected anatomical structures are in view. Piskun applies its own machine learning procedure to identify the structures, so the limitation Piskun leaves open is the qualifier that the detection is of structures in the field of view. The combination would have been obvious because a person of ordinary skill wanting Piskun's markers to report what the surgeon is currently looking at needed a detector that resolves anatomy against the current view, and Barral supplies that detector for the same endoscopic data. Each reference performs in the combination the same function it performs separately, since Barral's classifier supplies the detection result and Piskun's markers display it, so the result would have been predictable to one of ordinary skill.
Regarding claim 8, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, list of structures comprises a predetermined list of structures based on a type of surgery in the surgical video.
However, in a similar field of endeavor, Barral discloses wherein the list of structures comprises a predetermined list of structures based on a type of surgery in the surgical video (para 39, “Based on this received information, processing logic selects an ML medical procedure model from the data stored based on the specified medical procedure and configures the selected model based on the specified patient characteristics”; also, para 33, “In one embodiment, during the medical procedure, MLM based procedure engine 250 utilizes the classifiers of the selected ML medical procedure model to identify steps in a procedure, identify anatomical structures of a patient undergoing the procedure, and/or identify appropriate medical tool usage 260.”; also, para 41, “When the current operation in the medical procedure is a dissection operation (e.g., as identified by the classifiers of the selected/configured ML medical procedure model), an insert 604 can be displayed 606 based on the current operation to inform the surgeon what anatomical structures should be within the surgeon's field of view at the completion of the dissection (e.g., the triangle of Calot 608 surrounded by other anatomical structures such as the liver, arteries, ducts, etc.).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention with the features of Barral's invention of selecting a machine learning medical procedure model according to the medical procedure specified for the case, the classifiers of that selected model then identifying the anatomical structures of the patient undergoing that procedure and determining whether the anatomical structures expected for the current operation are in view. Because the anatomy classifier is a component of a model that is itself chosen by procedure, the anatomical structures that classifier is looking for are fixed by the procedure before the video is examined, which is what Barral's enumeration of the structures expected at the completion of a dissection illustrates. The combination would have been obvious because Piskun already draws its markers from a set established for the procedure before the operation begins, so tying the anatomical structures themselves to the procedure in the manner Barral teaches merely completes an arrangement Piskun already contemplates, and the result would have been predictable to one of ordinary skill.
Regarding claim 9, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, list of structures comprises a dynamic list of structures.
However, in a similar field of endeavor, Barral discloses wherein the list of structures comprises a dynamic list of structures (para 41, “When the current operation in the medical procedure is a dissection operation (e.g., as identified by the classifiers of the selected/configured ML medical procedure model), an insert 604 can be displayed 606 based on the current operation to inform the surgeon what anatomical structures should be within the surgeon's field of view at the completion of the dissection (e.g., the triangle of Calot 608 surrounded by other anatomical structures such as the liver, arteries, ducts, etc.).”; also, para 21, “The medical procedure system 140 analyzes the received sensor data with the ML medical procedure model to, for example, track the progress of the medical procedure, identify actions performed during the medical procedure, match identified actions to steps of a predefined sequence of steps for a given medical procedure, dynamically update a sequence of steps of a medical procedure given case-specific and/or action-specific information.”; as a surgeon continues its procedure such lists of structures will dynamically change based the surgeons current phase in operation to indicate different anatomical structures, dynamically presenting and displaying a list of structures).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher with the features of Barral's invention of presenting the anatomical structures that should be within the surgeon's field of view according to the current operation, the sequence of operations itself being dynamically updated as the procedure progresses. Because the set of anatomical structures Barral presents is keyed to the operation then under way, that set is not fixed for the whole case but changes as the procedure moves from one operation to the next. The combination would have been obvious because Piskun already changes what it presents to the surgeon as the procedure advances, and Barral supplies the corresponding treatment of the anatomical structures themselves, so a person of ordinary skill would have expected the predictable result of a list of structures that reflects the stage the operation has reached.
Regarding claim 13, Piskun as modified by Maegher discloses the computer-implemented method of claim 12, structure is one of an anatomical structure and a surgical instrument.
However, in a similar field of endeavor, Barral discloses wherein Piskun further discloses the structure is one of an anatomical structure and a surgical instrument (para 40, “As another example, the classifiers analyze visual sensor data, such as endoscope data, to recognize anatomical structures in a field of view to determine if the expected anatomical structures are in view, recognize which medical tools are currently being used, whether a deviation in the medical procedure has occurred due to the view of unexpected anatomical structures, etc.”; also, para 24, “In one embodiment, the medical procedure model includes multiple classifiers, such as a step identification classifier that is used to identify a current step in a medical procedure (e.g., current step is X, next step is Y with potential deviation of Z), an anatomy classifier that is used to identify one or more anatomical structures during a medical procedure based on input video or still image data (e.g., identification of a patient's arteries, organs, etc.), and one or more tool classifiers that identify proper and improper usage of a tool during a medical procedure based on medical tool sensor data (e.g., identification of an expected incision depth, tool movement(s) direction, speed, acceleration, etc.).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention of identifying a structure in surgical video and representing that structure with a marker, with the features of Barral's invention of classifiers that recognize anatomical structures in a field of view and recognize which medical tools are currently being used, the model including both an anatomy classifier and one or more tool classifiers. The combination would have been obvious because the video Piskun processes shows the instruments the surgeon is working with alongside the anatomy, and Barral teaches one classifier set that identifies both from that same operative data, so extending Piskun's identification to instruments required no change to how its markers are displayed. Each reference performs in the combination the function it performs separately, so the result would have been predictable to one of ordinary skill.
Regarding claim 14, Piskun as modified by Maegher and Barral discloses the computer-implemented method of claim 13, wherein Piskun further discloses the anatomical structure is one of an organ, artery, duct, surgical artifact, and anatomical landmark (para 30, “For example, the exemplary machine learning procedure can be used to predict (i) the cystic artery with a mask, (ii) the cystic duct with a mask, and/or (iii) the location of the gallbladder, the target zone, and the danger zone with masks.”; also, para 21, “The exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can be used to identify and/or track a surgical target zone, surgical danger zone and anatomical structures of significance, for example, cystic duct and cystic artery, in the subhepatic area of a patient's body on the display during laparoscopic gallbladder procedure.”).
Regarding claim 16, Piskun as modified by Maegher discloses the computer-implemented method of claim 12, structure is one from a predetermined list of structures.
However, in a similar field of endeavor, Barral discloses wherein the structure is one from a predetermined list of structures (para 29, “In one embodiment, data store 216 maintains a plurality of procedure and patient specific ML medical procedure models.”; also, para 40, “As another example, the classifiers analyze visual sensor data, such as endoscope data, to recognize anatomical structures in a field of view to determine if the expected anatomical structures are in view, recognize which medical tools are currently being used, whether a deviation in the medical procedure has occurred due to the view of unexpected anatomical structures, etc.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher with the features of Barral's invention of maintaining procedure specific machine learning medical procedure models and of classifiers that determine whether the expected anatomical structures are in view. The combination would have been obvious because an expectation set exists only if the structures to be looked for were established before the current video was examined, so Barral's expected anatomical structures are a predetermined set, and applying that arrangement to Piskun's identification step required no change to how Piskun detects structures or renders markers. The result would have been predictable to one of ordinary skill.
Claim(s) 4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piskun et al. (U.S. Pub. No. 20220093236) as modified by Meagher et al. (U.S. Pub. No. 20200297433), further in view of Mustufa et al. (U.S. Pub. No. 20190231459).
Regarding claim 4, Piskun as modified by Maegher discloses the computer-implemented user interface of claim 1, plurality of user interface elements is grouped in a single menu user interface.
However, in a similar field of endeavor, Mustufa discloses wherein the plurality of user interface elements is grouped in a single menu user interface (para 44, “The graphical menu 110 has a minimized configuration in which functions associated with instrument 26b are presented as small, minimized function icons 112a, 112b radially arranged about the periphery of a selector icon 114.”; also, para 42, “In one embodiment, as shown in FIG. 3, accessing features and capabilities for the instrument 26b in the field of view is enabled by a graphical menu 110 superimposed on the image 102.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher with the features of Mustufa's invention of a single graphical menu holding a plurality of icons together in one grouped arrangement on the surgical display. The combination would have been obvious because Piskun displays a plurality of markers together but does not organize them into any structure beyond the video overlay itself, and Mustufa teaches gathering the icons of a surgical display into one menu so the operator has a single place to look. A person of ordinary skill would have recognized that collecting the structure markers into one menu yields the predictable result that the surgeon reads the state of every tracked structure in one place, and that the markers no longer scatter across the image as the view changes.
Regarding claim 17, Piskun as modified by Maegher discloses the computer-implemented method of claims 12, wherein Piskun further discloses display of the video of the surgical procedure (para 22, “For example, one or more processors used with and/or by the exemplary system, method, and computer-accessible medium, according to an exemplary embodiment of the present disclosure, can receive a live video of a medical/surgical procedure (e.g., a laparoscopic surgical procedure), and in real time, provide the surgeon or another medical professional with an overlay over the video.”; also, para 52, “As shown in FIGS. 7-9, markers 705-720 can be provided as part of the video overlay.”). Piskun does not disclose wherein the user interface element comprises a geometric shape that is displayed at a predetermined position.
However, in a similar field of endeavor, Mustufa discloses wherein the user interface element comprises a geometric shape that is displayed at a predetermined position (para 57, “The menu includes a selector icon 206, which in this embodiment includes a partial circular shape with a pointer, and radially arranged function icons 208, 210.”; also, para 57, “In this embodiment, the menu 202 is in a fixed location in the surgical view, centered over a graphical image 204 of an instrument navigator representing a component of the teleoperational system.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher, in which markers are provided as part of the video overlay presented to the surgeon during the live video of the surgical procedure, with the features of Mustufa's invention of an icon drawn as a geometric shape and held at a fixed location in the surgical view. The combination would have been obvious because Piskun does not state what shape its markers take or where on the display they sit, and Mustufa teaches drawing such elements as simple geometric shapes at a set position over the surgical image. Each reference performs in the combination the function it performs separately, since Mustufa's teaching governs only the form and placement of the element and Piskun's marker continues to report structure state during the video, so a person of ordinary skill would have expected the predictable result that a simple geometric shape at a known position is read faster and is less likely to be confused with the anatomy behind it than an arbitrary graphic at a shifting position.
Claim(s) 5 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piskun et al. (U.S. Pub. No. 20220093236) as modified by Meagher et al. (U.S. Pub. No. 20200297433) and Mustufa et al. (U.S. Pub. No. 20190231459), further in view of Claus et al. (U.S. Pub. No. 20080316304).
Regarding claim 5, Piskun as modified by Meagher and Mustufa discloses the computer-implemented user interface of claim 4, wherein Piskun further discloses graphical overlay on the surgical video (para 22, “As shown in FIG. 1, one or more computer processors used with and/or by the exemplary system, method, and computer-accessible medium can be programmed to generate a video overlay which can highlight (e.g., using any color), and/or otherwise identify various anatomical structures in the video.”; also, para 52, “As shown in FIGS. 7-9, markers 705-720 can be provided as part of the video overlay.”). Piskun does not disclose the menu user interface is a toolbar.
However, in a similar field of endeavor, Claus discloses the menu user interface is a toolbar (para 58, “In Play mode, the present design may render a software toolbar to the video, for example located at the top or the bottom of the video presentation.”; also, para 28, “The present design is directed to providing customizable graphical video overlays superimposed over surgical procedure video recordings captured and recorded by a safety critical system such as a medical instrument system, for example a phacoemulsification/vitrectomy surgical system.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher and Mustufa, in which the grouped structure markers are provided as part of the video overlay on the surgical video, with the features of Claus's invention of a software toolbar rendered to the video at its top or bottom edge. The combination would have been obvious because Piskun as modified already places a grouped menu of structure markers over the surgical video without stating the form that grouping takes, and Claus teaches that in a surgical video display the grouped controls are rendered as a toolbar laid along an edge of the video. Each reference performs in the combination the function it performs separately, since Claus's teaching governs only the form and placement of the grouped elements and Piskun's markers continue to report structure state, so a person of ordinary skill would have expected the predictable result of keeping the operative view and the structure indicators within a single gaze while leaving the working area of the image uncovered.
Regarding claim 7, Piskun as modified by Meagher and Mustufa discloses the computer-implemented user interface of claim 5, location of rendering the toolbar is fixed or user-configurable.
However, in a similar field of endeavor, Claus discloses wherein a location of rendering the toolbar is fixed or user-configurable (para 58, “In Play mode, the present design may render a software toolbar to the video, for example located at the top or the bottom of the video presentation”; also, para 72, “During playback of the stored data, the user may configure and control the SMC system 201 to render the overlay data at a desired position for presentation on a video display device in association with, or superimposed on, or on top of the stored data presented, i.e. surgical procedure video, typically by using drag and drop or positional placement software functionality for a specific software module, such as desiring the aspiration pressure to be displayed in the upper right corner of the screen at position 155 horizontal, 0 vertical.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher and Mustufa with the features of Claus's invention of a toolbar placed at a set edge of the surgical video and of overlay elements the user positions on the display by drag and drop or by specifying screen coordinates. The combination would have been obvious because Piskun does not say where its markers are rendered, and Claus teaches both a fixed edge placement and a user-selected placement for material overlaid on surgical procedure video, either of which satisfies the recited disjunctive. A person of ordinary skill would have recognized that fixing the position of an indicator the surgeon must read repeatedly speeds reading, and that allowing the surgeon to move it prevents it from covering whatever part of the operative field matters at the moment, each being a predictable benefit taught by Claus.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Piskun et al. (U.S. Pub. No. 20220093236) as modified by Meagher et al. (U.S. Pub. No. 20200297433) and Barral et al. (U.S. Pub. No. 20180065248), further in view of Wolf et al. (U.S. Pub. No. 20200272660).
Regarding claim 15, Piskun as modified by Meagher and Barral discloses the computer-implemented method of claim 13, wherein the surgical instrument is one of clamps, staplers, knives, scalpels, sealers, dividers, dissectors, tissue fusion instruments, monopolars, Marylands, and fenestrated. Piskun as modified by Maegher and Barral does not disclose wherein the surgical instrument is one of clamps, staplers, knives, scalpels, sealers, dividers, dissectors, tissue fusion instruments, monopolars, Marylands, and fenestrated.
However, in a similar field of endeavor, Wolf discloses wherein the surgical instrument is one of clamps, staplers, knives, scalpels, sealers, dividers, dissectors, tissue fusion instruments, monopolars, Marylands, and fenestrated (para 102, “Instrument 301 is only one example of possible surgical instrument, and other surgical instruments such as scalpels, graspers (e.g., forceps), clamps and occluders, needles, retractors, cutters, dilators, suction tips, and tubes, sealing devices, irrigation and injection needles, scopes and probes, and the like, may include any suitable sensors and light-emitting sources.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Piskun's invention as modified by Meagher and Barral with the features of Wolf's invention of the surgical instruments used during a procedure, which Wolf enumerates as including scalpels and clamps, each of which is a recited species of the disjunctive list of the claim. The combination would have been obvious because Piskun tracks and marks objects appearing in the operative video and shows surgical tools and clips in that video without stating which instruments the procedure employs, and Wolf enumerates the instruments that appear in such video in the ordinary course of surgery. A person of ordinary skill would have recognized that the objects whose presence the interface reports are simply the anatomy and the instruments in use, and would have expected the predictable result that marking a scalpel or a clamp proceeds exactly as marking any other object the trained model was given.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights