DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 61, 68, and 75 are amended and claims 1-60 are cancelled. Claims 61-80 are pending.
Response to Arguments
Applicant's arguments filed 6/23/2026 have been fully considered.
Regarding the rejections of independent claims 61, 68, and 75 under 102, Examiner acknowledges as noted by Applicant that the amendments overcome the rejections, which are withdrawn. In view of further search and analysis of the prior arts, new grounds for rejecting claims 61, 68, and 75 under 103 are set forth herein.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 61, 64-68, 71-75, and 78-80 are rejected under 35 U.S.C. 103 as being unpatentable over Schroeder et al., "Exploratory Visualization of Surgical Training Databases for Improving Skill Acquisition," in IEEE Computer Graphics and Applications, vol. 32, no. 6, pp. 71-81, Nov.-Dec. 2012.
As to claims 61, 68, and 75, Schroeder teaches “[a] computer-implemented method for presenting surgical performance data (page 72, paragraph beginning with “Working toward these grand challenges” describing output of method for presenting surgical performance data, including use of multiview overlay and other visualization that require computer implementation; page 71, paragraph beginning with “Here, we explore visualization trends …” explaining application in computation environments),” “a non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method (page 71, paragraph beginning with “Here, we explore visualization trends …” explaining application in computation environments (would inherently and necessarily entail program instructions stored and executed for implementing the method)),” “a computer-system comprising: at least one processor; and at least one memory, the at least one memory comprising instructions configured to cause the computer system to perform a method for presenting surgical performance data (page 71, paragraph beginning with “Here, we explore visualization trends …” explaining application in computation environments (would inherently and necessarily entail processor and memory for storing and executing program instructions for implementing the method))”
the method comprising:
causing a first plurality of interface elements to be presented to a user in a first interface (page 72, paragraph beginning with “Working toward these grand challenges” describing multiview video overlay presentation; Figure 1 depicting the overlay view including multiple user interfaces (display objects) presented in an overall, overlayed display), the first plurality of interface elements forming a metric map in which each element of the first plurality of interface elements is a graphical representation of a respective surgical procedure (Figures 1(a) and 2(a) graphically depicting right and left grasp forces as points/lines in a mapping that constitutes a graphical representation of the laparoscopic procedure, and per Figure 2(a) subportions of the of the points/lines are selectable objects (interface elements)) associated with at least one surgical dataset of a first plurality of surgical datasets (Figure 1 overlay view includes display objects (a) (b) (c) (d) (e) presenting (associated with) multiple different aspects of surgical data in which “all views are linked” to a same set or subset of surgical data; pages 72-73, Our Visualization System, paragraphs beginning with “Inspired by …” and “All views are linked …”);
receiving, from the user, a selection of an interface element from the first plurality of interface elements (page 73, Our Visualization System, paragraph beginning with “All views are linked …” and page 73, Smart Brushing, paragraph beginning with “The basic interaction …” describing user selection of a portion of an interface element (e.g., portion of Figure 1 (a)); Figure 2(b) depicting selection of elements); and
in response, at least in part, to the user selection of the interface element from the first plurality of interface elements, causing a second interface to be presented to the user (pages 72-73, Our Visualization System, paragraphs beginning with “Inspired by …” and “All views are linked …” describing the multiple complementary interactive display objects in Figure 1 as being linked (referring to the same set of data in a time sequence (Figure 1 and Figure 1 caption depict temporal correspondence); page 73, paragraphs beginning with “All views are linked …” and “The basic interaction …” describing “smart brushing” in which user selection of a portion of an interface element (e.g., portion of Figure 1 (a)) results in a filtering and redisplay of the display objects that is limited to the portion selected of one of the interface elements; Figure 2(b) depicting an example of one of the second interface windows), the second interface comprising:
a playback region, the playback region depicting at least one video frame from the surgical dataset associated with the selected interface element (Figure 1 display objects (d) (e) provide video and 3-D trace over time. Examiner notes that per the smart brushing objects (d) (e) would be presented in correspondence with the user selection of the interface element (e.g., portion of Figure 1 (a)); and
a second plurality of interface elements, each interface element of the second plurality of interface elements being a graphical representation of the respective surgical procedure (Figure 2(b) and corresponding but not depicted transitions for objects in Figures 1(b)(c)(d)(e) would remain graphical representations of the surgical procedure) and associated with at least one surgical dataset of a second plurality of surgical datasets (Figure 1 display objects (a) (b) (c) provide surgical data that per the smart brushing would be presented in correspondence with the user selection of the interface element (e.g., portion of Figure 1 (a)).”
Regarding “wherein selection of an interface element of the second plurality of interface elements causes the second interface to be updated with the surgical dataset associated with the selected interface element of the second plurality of interface elements,” Schroeder does not explicitly describe/depict a further iteration of selecting/filtering interface elements that results in an updating of the second interface with the surgical dataset beyond a first iteration. However, Schroeder’s smart brushing technique as described on page 73, paragraphs beginning with “All views are linked …” and “The basic interaction …” describing “smart brushing” in which user selection of a portion of an interface element(s) results in a filtering and redisplay of the display objects is described in an open-ended manner such that is self-evidently and inferred as repeatable (e.g., the architecture comprising objects (a)(b)(c)(d)(f) in Figure 1 is maintained during smart brushing with only data content being modified/updated). Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date, to have implemented such an inferred extension of smart brushing in terms of a “re-selection” of portions of the data originally displayed in Figure 1(a) and redisplayed across objects (a)(b)(c)(d)(e) (depicted in part in Figure 2(b)) that would result in a further updating (display update based on re-selection) of the second interface based on object selection/filtering in the re-selection such as would constitute selection of an interface element of the second plurality of interface elements causes the second interface to be updated with the surgical dataset associated with the selected interface element of the second plurality of interface elements.
The motivation would have been to further refine the previously refined desired surgical related data for more user-optimized display to the user.
As to claims 64, 71, and 78, Schroeder teaches “wherein, both the first interface and the second interface present data associated with a same surgical task (the smart brushing technique depicted and described on pages 72-73 entails transitions/redisplay from a broader set to a narrower subset of the common data presented by the multiple, linked visualizations and therefore are associated with a same surgical task (e.g., surgical grasping depicted in Figure 1), and wherein, depicting at least one video frame from the surgical dataset associated with the selected interface element comprises advancing to a video frame associated with a starting time of the same surgical task (smart brushing results in redisplay of grasp force transitions over time, including in the mutually linked video visualization Figure 1(d), which would result in redisplaying the video visualization at a point corresponding to the user-selected points in Figure 2(a) (temporally associated as indicated in Figure 2 caption). Examiner notes that the new restart position in the video would constitute a starting time of a surgical task/subtask delimited by the new start/end boundaries).”
As to claims 65, 72, and 79, Schroeder teaches “wherein, the second interface depicts a plurality of metric values of a same type of metric (Schroeder: redisplaying the visualizations in Figure 1 via smart brushing are linked (same data set as per Figure 1 caption) and may depict multiple values of a same metric type (e.g., multiple values of grasp force as depicted in Figure 1 (a) (c) (d) and (e)), the metric values associated with each surgical dataset of the second plurality of surgical datasets (the presented visualizations are necessarily associated (generated from) underlying data corresponding to the different presentations), and wherein, the second interface depicts a time of acquisition associated with each surgical dataset of the second plurality of surgical datasets (the redisplayed version of Figure 1 (c) (depicting time series variation of grasp force showing times corresponding to determined grasp forces (acquisition times)) would display acquisition times corresponding to each of the temporally linked visualizations. Figure (e) also depicted the grasp force associated times).”
As to claims 66, 73, and 80, Schroeder appears to suggest in the description on pages 72-73 and on page 77, caption for Figure 5, and page 78, Scalability and the Need for Good Data Models, paragraph beginning with “This approach also supports …”, that the smart brushing technique depicted and described on pages 72-73 in which a first interface element is selected to cause redisplay of the visualization elements in Figure 1 delimited by the user-selected element is applicable in a repeatable, iterative manner (an interface element (e.g., Figures 1(a) and 2(a) in a redisplayed, delimited multi-view presentation may be selected to repeat the process).
Moreover, it would have been obvious to one of ordinary skill in the art before the filing date, to have applied the smart brushing technique in a second iteration given that the first smart brushing iteration results in the same set of visualization imaging objects that can then be reselected in the same manner using the same data to produce analogous results. In such configuration, the method would include “in response, at least in part, to a user selection of an interface element of the second plurality of interface elements, causing data of the surgical dataset associated with the selected interface element of the second plurality of interface elements to appear in the playback region; and advancing the playback region to a video frame associated with a starting time of the same surgical task, and wherein, the first plurality of surgical datasets and the second plurality of surgical datasets are the same plurality of surgical datasets,” as set forth in the grounds for rejecting claims 64-65.
The motivation for such a configuration would be the same as for an initial iteration of the smart brushing process – to further select particular surgical data of interest from an applicable interface element of an initial interface for redisplaying surgical data within the overall same set of data and delimited via the selection, and the potential for successfully implementing such second iteration would have been readily appreciated by one of ordinary skill since it is simply a repeat of an initial iteration using the same data constructs and methodology.
As to claims 67 and 74, it is self-evident that the metric data presented via Schroeder’s method (e.g., on-scene vide in Figure 1 (d) and grasp force visualizations in Figure 1 (a) (c) and (e)) can only be obtained by some form of on-scene (i.e., within a real or simulated surgical theater/environment such as a video camera and/or grasp forces sensors). Furthermore, Schroeder discloses on page 74 that surgical theater sensor platforms are known in the art for obtaining the surgical data to be presented (page 74 FIG. A and associate description).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Schroeder’s disclosure of known on-site sensor-based collection of data to be presented to Schroeder’s effective teaching of the need for sensor-based, on-site data collection, such that in combination the method includes deriving each metric value of the plurality of metric values of the same type of metric, at least in part, from sensor data acquired from sensors present in a surgical theater.
The motivation would have been to provide the underlying metric data to enable presentation of the data as suggested by Schroeder.
Claims 62, 69, and 76 are rejected under 35 U.S.C. 103 as being unpatentable over Schroeder as applied to claims 61, 68, and 75 above, and further in view of El-Saig et al., "A Graphical Tool for Parsing and Inspecting Surgical Robotic Datasets," 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), Budapest, Hungary, 2018.
As to claims 62, 69, and 76, Schroeder teaches “wherein,” “the second plurality of interface elements comprises a plurality of scatter plot points in a scatter plot of the second interface (Figure 1 (a) and Figure 2 (a) display object that is displayed and redisplayed via smart brushing entails 2-D plot points showing trends between different metrics (scatter plot)).”
Schroeder teaches various interface elements for displaying surgical data in various ways in which the various data are logically inter-related (Figure 1), but does not appear to teach that the first interface includes tabular data including multiple rows and therefore does not teach “the first plurality of interface elements comprises a plurality of rows in a table of the first interface.”
Displaying surgical data in a user interface using tabular data including multiple row was known in the art prior to the effective filing date. For example, El-Saig discloses a method for presenting surgical data in an orderly manner (Abstract; page 2, C. Design, paragraph beginning with “Due to the preferred visual nature …”) that includes a user interface in which surgical data is presented, in part, as multiple rows in a table of an interface (Fig. 4 (lower picture) interface includes a table object “Descriptor” including multiple rows).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied El-Saig’s teaching of presenting multiple rows in a table of a user interface to the method taught by Schroeder, which teaches a first interface comprising a first plurality of interface elements in which selection of one of the interface elements may be used to transition to a second interface, such that in combination the method includes “the first plurality of interface elements comprises a plurality of rows in a table of the first interface.”
The motivation would have been to provide human-readable associations of data within the interface to enhance a user’s ability to understand particular data associations (e.g., between video or other non-textual data presented among the interface elements) relevant to efficient retrieval/parsing of stored surgical data as suggested by El-Saig.
Claims 63, 70, and 77 are rejected under 35 U.S.C. 103 as being unpatentable over Schroeder in view of El-Saig as applied to claims 62, 69, and 76 above, and further in view of Lendvay (US 2015/0044654 A1).
As to claims 63, 70, and 77 the combination of Schroeder and El-Saig teaches “wherein,”
“the scatter plot comprises:
a first axis indicating a range of metric values associated with a single type of metric of the at least two types of metrics (Schroeder: Figures 1(a) and 2(a) depicting a first axis (y-axis) indicating a range of Right Hand Grasp Force and a second axis (x-axis) indicating a range of Left Hand Grasp Force);”
“and wherein, the method further comprises:
in response, at least in part, to a user selection of a scatter plot point, updating the playback region with data of the surgical dataset associated with the selected scatter plot point (Schroeder: page 73, paragraphs beginning with “All views are linked …” and “The basic interaction …” describing “smart brushing” in which user selection of a portion of an interface element (e.g., portion of Figure 1 (a) – scatter plot) results in a filtering and redisplay of the display objects including redisplay/update of Figure 1 (d) (video) that is limited to the portion selected of one of the interface elements).”
El-Saig further teaches “each row of the plurality of rows depicts metric values of at least two types of metrics for” [a] “surgical dataset associated with the row (Fig. 4 lower picture, depicting table object “Descriptor” including multiple rows in which each row depicts START and END times (two types of metrics) for various surgical tasks).”
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied El-Saig’s teaching of presenting surgical task related information in a table in which each row depicts two types of metrics to the method taught by Schroeder as modified by El-Saig in which the first interface includes a table with multiple rows, such that in combination the method includes configuring the table such that each row of the plurality of rows depicts metric values of at least two types of metrics for the surgical dataset associated with the row.
The motivation would have been to provide multiple forms of metric data to further enable efficient comprehension of a user as to the data that is or may be selected to be presented in the first or second interface as suggested by El-Saig.
Regarding the scatter plot configuration as including a first axis indicating a range of metric values associated with a single type of metric of the at least two types of metrics “and a second axis indicating acquisition times of each surgical dataset of the second plurality of surgical datasets,” Schroeder discloses that the user interface (e.g., Figure 1) presents surgical data in various visualization formats including a scatter plot (Figures 1(a) and 2(a)) in which each of the visualization formats is associated with and synchronized in time (Figure 1 caption visualizations are linked, temporally associated, and refer to same data set; Figure 2 caption explaining that user selection of portions of the left and right grasp force plot corresponds to associated times (acquisition times)). However, Schroeder does not expressly teach that the temporal variation of the surgical dataset (e.g., metric) may be explicitly conveyed via a “second axis indicating acquisition times of each surgical dataset of the second plurality of surgical datasets.”
Prior to the effective filing date, it was known that a time-based scatter plot (i.e., scatter plot including a dependent axis illustrating variations of a metric and an independent time axis) is a form of visualization interface that may be used to present surgical related information. For example, Lendvay discloses a method for evaluating surgical skills (Abstract) that includes presenting various forms of data visualization interfaces including a scatter plot (FIG. 10 plot of evaluation group participation includes discrete intersection points and therefore constitutes scatter plot) having a first axis indicating a metric (percent participation) and a second axis indicating corresponding times (FIG. 10 plot correlates percent participation with corresponding times).
It would have been obvious to one of ordinary skill in the art before the effective filing date, to have applied Lendvay’s teaching of using time-based scatter plots to present time-varying data to the method taught by the combination of Schroeder and El-Saig in which surgical data metric acquisition times are used as ranges over which surgical data is presented (e.g., Schroeder Figure 1) and in which scatter plots are used as visualization interfaces for surgical task metrics (e.g., Schroeder Figures 1(a) and 2(a)), such that in combination the method includes a second interface having a scatter plot that includes a first axis indicating a range of metric values associated with a single type of metric of the at least two types of metrics and a second axis indicating acquisition times of each surgical dataset of the second plurality of surgical datasets.
The motivation would have been to provide an alternative and/or additional visual presentation technique for depicting variations in metric values (e.g., grasp force) over time to more comprehensively present such temporal metric variations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW W BACA whose telephone number is (571)272-2507. The examiner can normally be reached Monday - Friday 8:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Schechter can be reached at (571) 272-2302. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW W. BACA/Examiner, Art Unit 2857
/ALEXANDER SATANOVSKY/Primary Examiner, Art Unit 2857
Prosecution Insights