DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The claims were amended by Preliminary Amendment filed 3/11/2025, canceling claim 1 and adding new claims 2-21. As such claims 2-21 are currently pending and under consideration.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 15, the claim recites, “wherein the instructions to display comprises instructions to display a virtual marking based on the input superimposed over the region of interest of the patient and the virtual representation of the remote user” in lines 6-7. As currently drafted, the claim first is unclear whether the display of a virtual marker is “superimposed over the region of interest” or based on input, where the input is superimposed over the region of interest. The most logical interpretation is that the virtual marking is superimposed, and not the input. Second, the claim is drafted in a manner that it is unclear what the “instructions to display” relate to, as the may merely display a virtual marking (where the virtual marking is displayed superimposed over the region of interest of the patient and the virtual marker is displayed over the virtual representation of the remote user) or the instructions display both a virtual marker (based on the input) superimposed over the region of interest and the instructions also display the virtual representation of the remote user. As such, the claim is rendered indefinite.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-5, 8-10, 12 and 15-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Johnson et al. (US 2019/0254754 A1) in view of
Anderson et al. (US 2021/0072947 A1).
Regarding claim 10, Johnson et al. (US 2019/0254754 A1)
A surgical robotic system (Johnson: Abstract and ¶8: augmented reality navigation system for use with a robotic surgical system), comprising:
a surgical robot (Johnson, ¶8: robotic surgical system (e.g., a robotic arm, a part of a robotic arm, and/or an end-effector of a robotic arm);
a first portable electronic device that comprises a screen and a first camera configured to capture a first video stream and operated by a local user (Johnson, ¶81: AR surgical system including HMD, including display screen; ¶85: detector camera mounted to HMD);
a second camera that is separate from the first portable electronic device and is arranged within an operating room (OR), and that is configured to capture a second video stream that includes at least one of a patient, the surgical robot, or the first portable electronic device within the OR (Johnson, ¶91; auxiliary detector in surgical environment in addition to detector disposed on HMD; ¶92: second HMD with detector; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶131: at least one detector tracks real-world features identifying or attached to patient and robotic surgical system, where the at least one detector 902 includes a plurality of cameras that are spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked);
a second portable electronic device that is located at a location other than the OR and is operated by a remote user (Johnson, ¶51 and ¶92: video input feed is provided to an external monitor (e.g., on a nearby wall or in a remote location, such as a class room) for view by other persons (e.g., in the same surgical environment), either in addition to or in alternative to being provided to a second augmented reality navigation system);
at least one processor (Johnson, ¶8: processor of computing device; Also ¶163); and
memory having instructions which when executed by the at least one processor causes the surgical robotic system to perform the following operations: (Johnson, ¶8: non-transitory computer readable medium having instructions stored and executed by processor; Also ¶163)
generate a virtual OR as a virtual representation of the OR that includes the patient and the surgical robot and establishing a common coordinate frame between the virtual OR, the surgical robot, and the first portable electronic device based on the first video stream and the second video stream (Johnson, ¶111: computer equipment receives plurality of video streams and information; ¶124: position tracking system 810 including detectors mounted on HMD and cameras spaced apart in operating room, to track location of patient’s anatomy, surgical tool and surgical apparatus, and generate 3D model of targeted site of patient; ¶125: model can include real-world features or registrations between virtual locations of the representations of real-world features in the patient model and physical locations of the real world features on patient’s body with precise overlay; ¶126: augmentation graphics may be portions of surgical tool or apparatus; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶133: relative positioning module 916 identifies relative position of each tracked real-world features 904-910 – see Fig. 11, which includes patent and robot, as disclosed in ¶131 – and module performs coordinate transformations of relative coordinate systems of surgical table, patient, surgical tool, and HMD to unified, common coordinate system, where robotic surgical system coordinates data and tool coordinates data are equivalent; ¶136: portion of surgical tool and robotic surgical system provided as graphical overlay; ¶139 discloses display of AR graphics; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system – i.e. virtual OR; also ¶147: unified coordinate system resulting from registration of augmented reality navigation system and surgical environment, including objects in surgical environment including patient and robotic surgical system)
Johnson further teaches a plurality of portable devices, such as HMDs and coordinating the virtual positions between the plurality of portable devices using a common coordinate frame (Johnson, ¶92: two surgeons wearing HMDs with a shared a common registration).
Johnson, however, does not explicitly teach the second device as portable and further incorporation of a remote user virtual representation in the augmented reality surgical system as claimed. Collaborative augmented reality systems that coordinate a remote user within an augmented reality system of another user, however, was known before the effective filing date of the claimed invention.
Anderson discloses:
a second portable electronic device that is located at a location other than the [first location] and is operated by a remote user (Anderson, Fig. 1 and ¶17: user/learner and instructor, remotely located; ¶¶20-21: both user/learner and instructor use head-mounted AR/VR display)
receive, from the second portable electronic device, a virtual representation of the remote user who is in the location, wherein the virtual representation of the remote user is at a virtual position in the common coordinate frame (Anderson, ¶19: capturing 3D spatial data in both locations; ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶26: tacking body part of users; ¶27: 3D location information received from instructor room, and generates AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; also ¶29: synchronizing the location of objects in shared 3D virtual space 103; ¶42: included in AR view a synchronous avatar of remote user with real-time location and orientation of position tracked object oof remote user, e.g. hands and head), and
display, on the screen, the virtual representation of the remote user at a position within the OR that corresponds with the virtual position in the common coordinate frame (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 2, the system of claim 10 performs the method of claim 2 and as such claim 2 is rejected based on the same rationale as claim 10 set forth above.
Regarding claim 3, Johnson further discloses:
Wherein the second video stream comprises at least a portion of the patient and the OR, (Johnson, ¶54: real world features detected by detector including spatial information about an object, such as operating table, robotic surgical system, a patient and objects in surgical environment; ¶92: HMDs having mounted detectors; ¶99: video streams from HMD camera capturing user line-of-sight) wherein displaying comprises displaying the second video stream with a region of interest based on input (Johnson, ¶10: the anatomical model augmentation graphics such that the updated anatomical model appears overlaid on the anatomy of the patient; ¶114: HMD including detector and processing video signal from detector augmented by augmentation graphics displayed on a display screen, where augmentation graphics comprising one or more symbols may, for example, identify the first wearer as the source of the video stream and/or be added to a video stream by a wearer to identify observed features, such as a patient's anatomy; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy); Fig. 11 and ¶138: augmented graphics displayed including trajectory of surgical tool, apparatus, or robotic surgical system; ¶142: a position of a surgical tool and/or its trajectory can be visualized over a period of time using augmentation graphics such that a surgeon can watch how a tool will move along a trajectory during at least a portion of a procedure; Also ¶143 and ¶149)
Johnson modified by Anderson further discloses:
wherein displaying comprises displaying the second video stream with a region of interest based on input from the remote user, (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶22: graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations; Also ¶49 discloses user may see other participant’s remote space to observe, inspect and comment on the remote user’s body movements and interactions with tools or objects, and as remote user performs task within remote user’s own environment; Also ¶50: remote user situated in local space with AR as if remote user were in local space; Fig. 1 and ¶17: user/learner and instructor, remotely located; ¶¶20-21: both user/learner and instructor use head-mounted AR/VR display) and the virtual representation of the remote user at the position within the room (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 4, Johnson further discloses:
wherein the region of interest comprises a virtual marking of at least one of an anatomy of the patient, an annotation by the remote user, or a location on the patient for an entry point for a surgical robotic instrument of the surgical robot. (Johnson, ¶10: the anatomical model augmentation graphics such that the updated anatomical model appears overlaid on the anatomy of the patient; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy); Fig. 11 and ¶138: augmented graphics displayed including trajectory of surgical tool, apparatus, or robotic surgical system; ¶142: a position of a surgical tool and/or its trajectory can be visualized over a period of time using augmentation graphics such that a surgeon can watch how a tool will move along a trajectory during at least a portion of a procedure; Also ¶143 and ¶149)
Regarding claim 5, Johnson further discloses:
The position within the OR is in a field of view of the second camera of the portable electronic device (Johnson, ¶91; auxiliary detector in surgical environment in addition to detector disposed on HMD; ¶92: second HMD with detector; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶131: at least one detector tracks real-world features identifying or attached to patient and robotic surgical system, where the at least one detector 902 includes a plurality of cameras that are spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked)
Johnson modified by Anderson further discloses:
Wherein the virtual representation of the remote user is only displayed on the screen of the portable electronic device, while the position within the room is in a field of view of the second camera of the portable electronic device (Anderson, ¶21: see-through AR projection system; ¶22: When head-mounted AR/VR display 150 is in AR mode, graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150. Thus, the user's view of the physical world is augmented with additional graphics, where such graphics include synchronous avatar 101A or 102A (which corresponds to the location of the remote user in 3D shared virtual space 103); Also ¶20, ¶27, ¶29, ¶42 discusses coordinating AR; Fig. 1 and ¶33: the head-mounted AR/VR display 150 of instructor room 110 includes an AR camera 252A, which is positioned on head-mounted AR/VR display 150 to capture real-time 3D video content from the current actual point of view of instructor 102 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 8, Johnson further discloses:
Wherein the portable electronic device is a (Johnson, ¶6: at least partially transparent display screen; ¶95: see-through display allowing user to see video superimposed on environment seen through the display screen; ¶124: AR for operating room; Fig. 11 and ¶131)
Johnson modified by Anderson further discloses:
Wherein the portable electronic device is a virtual reality headset worn by the local user, and the screen is a transparent display onto which the virtual representation of the remote user is displayed and through which the room is viewable by the local user (Anderson, ¶21: see-through AR projection system; ¶22: When head-mounted AR/VR display 150 is in AR mode, graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150. Thus, the user's view of the physical world is augmented with additional graphics, where such graphics include synchronous avatar 101A or 102A (which corresponds to the location of the remote user in 3D shared virtual space 103); Also ¶20, ¶27, ¶29, ¶42 discusses coordinating AR; Fig. 1 and ¶33: the head-mounted AR/VR display 150 of instructor room 110 includes an AR camera 252A, which is positioned on head-mounted AR/VR display 150 to capture real-time 3D video content from the current actual point of view of instructor 102 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 9, Johnson further discloses:
wherein the generating of the virtual OR and the displaying of the virtual representation (Johnson, ¶111: computer equipment receives plurality of video streams and information; ¶124: position tracking system 810 including detectors mounted on HMD and cameras spaced apart in operating room, to track location of patient’s anatomy, surgical tool and surgical apparatus, and generate 3D model of targeted site of patient; ¶125: model can include real-world features or registrations between virtual locations of the representations of real-world features in the patient model and physical locations of the real world features on patient’s body with precise overlay; ¶126: augmentation graphics may be portions of surgical tool or apparatus; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶133: relative positioning module 916 identifies relative position of each tracked real-world features 904-910 – see Fig. 11, which includes patent and robot, as disclosed in ¶131 – and module performs coordinate transformations of relative coordinate systems of surgical table, patient, surgical tool, and HMD to unified, common coordinate system, where robotic surgical system coordinates data and tool coordinates data are equivalent; ¶136: portion of surgical tool and robotic surgical system provided as graphical overlay; ¶139 discloses display of AR graphics; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system – i.e. virtual OR; also ¶147: unified coordinate system resulting from registration of augmented reality navigation system and surgical environment, including objects in surgical environment including patient and robotic surgical system)
Johnson modified by Anderson further discloses:
wherein the generating of the virtual OR and the displaying of the virtual representation of the remote user are performed continuously and contemporaneously with the receiving of the first video stream and the second video stream (Anderson, ¶20: real-time synchronous avatars in shared 3D space; ¶23: streaming and displaying synchronous content in real-time; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations; Also ¶49 discloses user may see other participant’s remote space to observe, inspect and comment on the remote user’s body movements and interactions with tools or objects, and as remote user performs task within remote user’s own environment; Also ¶50: remote user situated in local space with AR as if remote user were in local space; Fig. 1 and ¶17: user/learner and instructor, remotely located)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 12, Johnson modified by Anderson further discloses:
wherein the operations are performed while the first portable electronic device, the surgical robot, and the second camera are arranged within the OR, and during a surgical procedure by the surgical robot on the patient (Examiner notes this limitation appears to be intended use, as it merely states the system is used in a particular environment that does nothing to change the function of the claimed invention; However, Johnson, ¶81: AR surgical system including HMD, including display screen, worn by surgeon during a medical procedure; ¶85: detector camera mounted to HMD ¶91; auxiliary detector in surgical environment in addition to detector disposed on HMD; ¶92: second HMD with detector; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶131: at least one detector tracks real-world features identifying or attached to patient and robotic surgical system, where the at least one detector 902 includes a plurality of cameras that are spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked; ¶135: portion of patient’s body within field of view of what surgeon sees through display screen 110; ¶137: semitransparent display screen)
Regarding claim 15, Johnson further discloses:
Wherein the second portable electronic device is configured to receive input from the (Johnson, ¶92 discloses two surgeons in shared common registration wearing HMDs during surgical operation; ¶99: video streams from HMD; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy), and trigger capture of data that represents a position and/or orientation (e.g., of a trajectory) of interest that is indicated by the trajectory selection guidance augmentation graphic; ¶125 discloses modeling patient and physical locations in real-world features such that augmentation graphical overlay is precisely orientated and scaled on surgical site)
wherein the instructions to display comprises instructions to display a virtual marking based on the input superimposed over the region of interest of the patient (Johnson, ¶23: the plurality of surgical tool augmentation graphics such that the surgical tool augmentation graphics appear overlaid on the anatomy of the patient; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy), and trigger capture of data that represents a position and/or orientation (e.g., of a trajectory) of interest that is indicated by the trajectory selection guidance augmentation graphic; ¶125 discloses modeling patient and physical locations in real-world features such that augmentation graphical overlay is precisely orientated and scaled on surgical site)
Johnson modified by Anderson further discloses:
wherein the second portable electronic device is configured to receive input from the remote user associated with a region of interest (Anderson, ¶18: remotely located users in shared virtual 3D space; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations; Also ¶49 discloses user may see other participant’s remote space to observe, inspect and comment on the remote user’s body movements and interactions with tools or objects, and as remote user performs task within remote user’s own environment; Also ¶50: remote user situated in local space with AR as if remote user were in local space; Fig. 1 and ¶17: user/learner and instructor, remotely located; ¶¶20-21: both user/learner and instructor use head-mounted AR/VR display)
wherein the instructions to display comprises instructions to display a virtual marking based on the input superimposed over the region of interest (Anderson, ¶22: graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations) and the virtual representation of the remote user (Note claim interpreted as instructions displaying both virtual marking and displaying virtual representation despite 112(b) issue; Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 16, Johnson further discloses:
wherein the surgical robot comprises a robotic arm (Johnson, ¶12: robotic arm of robotic surgical system; ¶22: AR system with use of robotic surgical system with robotic arm) wherein the memory has further instructions to: receive, (Johnson discloses the operation of a collaborative augmented reality system including receiving way points created by the surgeon user for guiding and orientating a robotic arm relative to a patient during a surgical operation, as follows:
¶23: surgical tool trajectory, where trajectory augmentation graphics indicate a physical trajectory that could be followed during the surgical procedure; ¶26: determine a selected trajectory based at least in part on a user input trajectory selection signal, the user input trajectory selection signal corresponds to a gesture by the user or position/orientation of robotic arm; ¶¶26-27: robotic arm moved along trajectory; ¶123: trajectory selection guidance augmentation graphics positioned within plane of video stream by surgeon and aligned with point of interest; ¶126: augmentation graphics may be a surgical tool or apparatus, where computer subsystem can animate movement of a displayed patient mode, tool, or surgical apparatus to illustrate a planned procedure relative to a defined surgical site or target location on patient’s body; ¶139: an augmented reality navigation system is used in conjunction with a pointer tool in order to define a trajectory that a robotic surgical system can follow; ¶149: output from the augmented reality navigation system causes the robotic surgical system to automatically align with the represented trajectory and/or automatically move along the trajectory; ¶154: trajectory determined and updated based on position or orientation of pointer tool, which is used to select a plurality of points in space that are then used to collectively determine a trajectory, in response to user input signal received by computer subsystem, and trajectory is output to robotic surgical system for performing surgical procedure; ¶155: output from the augmented reality navigation system causes the robotic surgical system to automatically align with the represented trajectory and/or automatically move along the trajectory, where trajectory can be overlayed graphic on display screen
Johnson further discloses a collaborative environment and allowing control over the surgical robot remotely, which provides the technical functionality required for the claimed invention – See ¶56: robot configured to be manipulated remotely by surgeon; ¶92: two users of separate head mounted displays in co-registered AR navigation systems used in single surgical environment); and
display, on the screen, virtual representations of the one or more waypoints (Johnson, ¶20: representation of a trajectory of a surgical tool and update augmented reality graphics based on the modification of the representation; ¶22: computer subsystem configured to generate and/or access a representation of at least a portion of a surgical tool and/or a trajectory of the surgical tool during a surgical procedure, modify at least a portion of the representation based on a relative position and/or orientation of one or more real-world features in a detector input signal received from the at least one detector, and display, on the display screen, surgical tool augmentation graphics based on the modified representation, wherein the surgical tool is inserted into or connected to the robotic surgical system; ¶114: video of procedure provided as real-time video stream to other HMDs worn by personnel assisting with procedure, where video sign is augmented to comprise both video signal and one or more symbols; Fig. 11 and ¶138: augmented graphics displayed including trajectory of surgical tool, apparatus, or robotic surgical system; ¶142: a position of a surgical tool and/or its trajectory can be visualized over a period of time using augmentation graphics such that a surgeon can watch how a tool will move along a trajectory during at least a portion of a procedure; Also ¶143 and ¶149)
Even though Johnson provides a system remotely operable by a surgeon and providing input from a surgeon to guide a robotic arm for a procedure, and displaying navigational guidance information for the trajectory and location of the robotic device as an augmented reality graphic registered with a surgical device, Johnson does not explicitly tie the creation of the waypoint guidance to the remote user as claimed.
Anderson teaches:
receive, from the second portable electronic device, one or more waypoints created by the remote user for guiding an activity within the real space of another user (Anderson, ¶18: remotely located users in shared virtual 3D space; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations; Also ¶49 discloses user may see other participant’s remote space to observe, inspect and comment on the remote user’s body movements and interactions with tools or objects, and as remote user performs task within remote user’s own environment; Also ¶50: remote user situated in local space with AR as if remote user were in local space; Fig. 1 and ¶17: user/learner and instructor, remotely located; ¶¶20-21: both user/learner and instructor use head-mounted AR/VR display)
It is the combination of the teachings of Johnson, which teaches the inputting of navigation data by a user for guiding a robotic arm that includes a displayed waypoint and within the collaborative environment, which also has the ability for remote user interactivity during a surgical procedure as provided by Johnson, combined with the known technique of a remote user providing the input to provide displayed annotated guidance within a shared interactive augmented reality system that teaches the limitations of the claim.
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 17, Johnson discloses:
A method performed by at least one programmed processor of a first portable electronic device, the method comprising: Johnson: Abstract and ¶8: augmented reality navigation system including head mounted display for use with a robotic surgical system including processor of computing device executing instructions)
receiving a virtual representation of an operating room (OR) that includes a patient and a surgical robot, wherein the virtual representation of the OR comprises a common coordinate frame with at least the surgical robot; (Johnson, ¶111: computer equipment receives plurality of video streams and information; ¶124: position tracking system 810 including detectors mounted on HMD and cameras spaced apart in operating room, to track location of patient’s anatomy, surgical tool and surgical apparatus, and generate 3D model of targeted site of patient; ¶125: model can include real-world features or registrations between virtual locations of the representations of real-world features in the patient model and physical locations of the real world features on patient’s body with precise overlay; ¶126: augmentation graphics may be portions of surgical tool or apparatus; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶133: relative positioning module 916 identifies relative position of each tracked real-world features 904-910 – see Fig. 11, which includes patent and robot, as disclosed in ¶131 – and module performs coordinate transformations of relative coordinate systems of surgical table, patient, surgical tool, and HMD to unified, common coordinate system, where robotic surgical system coordinates data and tool coordinates data are equivalent; ¶136: portion of surgical tool and robotic surgical system provided as graphical overlay, as well as parts of patient such as patient model of bone aligned with patient during surgery; ¶139 discloses display of AR graphics; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system – i.e. virtual OR; also ¶147: unified coordinate system resulting from registration of augmented reality navigation system and surgical environment, including objects in surgical environment including patient and robotic surgical system)
determining, based on sensor data captured by a sensor of the first portable electronic device, a location and an orientation of a user of the first portable electronic device, wherein the location and the orientation are in relation to the common coordinate frame (Johnson, ¶54: real world features detected by detector including spatial information about an object, such as operating table, robotic surgical system, a patient and objects in surgical environment; ¶92: HMDs having mounted detectors; ¶99: video streams from HMD camera capturing user line-of-sight; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶133: relative positioning module 916 identifies relative position of each tracked real-world features 904-910 – see Fig. 11, which includes patent and robot, as disclosed in ¶131 – and module performs coordinate transformations of relative coordinate systems of surgical table, patient, surgical tool, and HMD to unified, common coordinate system, where robotic surgical system coordinates data and tool coordinates data are equivalent; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system – i.e. virtual OR; also ¶147: unified coordinate system resulting from registration of augmented reality navigation system and surgical environment, including objects in surgical environment including patient and robotic surgical system)
displaying, on a screen of the first portable electronic device, the virtual representation of the OR (Johnson, ¶10: the anatomical model augmentation graphics such that the updated anatomical model appears overlaid on the anatomy of the patient; ¶136: portion of surgical tool and robotic surgical system provided as graphical overlay; ¶139 discloses display of AR graphics; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy); Fig. 11 and ¶138: augmented graphics displayed including trajectory of surgical tool, apparatus, or robotic surgical system; ¶142: a position of a surgical tool and/or its trajectory can be visualized over a period of time using augmentation graphics such that a surgeon can watch how a tool will move along a trajectory during at least a portion of a procedure; Also ¶143 and ¶149)
receiving input to include a virtual marking or a virtual object within the virtual representation of the OR (Johnson, ¶10: the anatomical model augmentation graphics such that the updated anatomical model appears overlaid on the anatomy of the patient; ¶114: HMD including detector and processing video signal from detector augmented by augmentation graphics displayed on a display screen, where augmentation graphics comprising one or more symbols may, for example, identify the first wearer as the source of the video stream and/or be added to a video stream by a wearer to identify observed features, such as a patient's anatomy; ¶123: a surgeon may orient his or her natural field of view to a surgical site, steer a trajectory selection guidance augmentation graphic to be aligned with a point-of-interest (e.g., in a patient's anatomy)); and
Anderson discloses:
transmitting, to a second portable electronic device, the location and orientation of the user and the input to render a virtual representation of the user and the virtual marking or the virtual object in augmented reality to a screen of the second portable electronic device (Anderson, Fig. 1 and ¶17: user/learner and instructor, remotely located; ¶¶20-21 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶22: graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space; ¶44: instructor and learner can both see annotations made in shared 3D virtual space, where instructor can annotate points of interest; ¶45: guidance mode, where instructor can provide proactive cues, guidance and other real-time feedback regarding performance of learner using annotations; Also ¶49 discloses user may see other participant’s remote space to observe, inspect and comment on the remote user’s body movements and interactions with tools or objects, and as remote user performs task within remote user’s own environment; Also ¶50: remote user situated in local space with AR as if remote user were in local space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Regarding claim 18, Johnson further discloses:
Wherein the virtual representation of the OR comprises a virtual representation of the patient, wherein the virtual marking or the virtual object is in relation to an anatomical feature of the virtual representation of the patient (Johnson, ¶136: portion of surgical tool and robotic surgical system provided as graphical overlay, as well as parts of patient such as patient model of bone aligned with patient during surgery; ¶139 discloses display of AR graphics; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system, and at least a portion of a surgical tool, at least a portion of a surgical apparatus, and/or at least a portion of a robotic surgical system (e.g., that is covered by a patient's anatomy) can appear as a graphical overlay matching the orientation and size of the physical object to the surgeon using augmentation graphics, in some embodiments.)
Regarding claim 19, Johnson further discloses:
Wherein the virtual representation of the OR comprises a virtual representation of a robotic arm of the surgical robot, wherein the virtual object comprises a waypoint for guiding and orientation the robotic arm relative to the patient in the OR (¶23: surgical tool trajectory, where trajectory augmentation graphics indicate a physical trajectory that could be followed during the surgical procedure; ¶26: determine a selected trajectory based at least in part on a user input trajectory selection signal, the user input trajectory selection signal corresponds to a gesture by the user or position/orientation of robotic arm; ¶¶26-27: robotic arm moved along trajectory; ¶123: trajectory selection guidance augmentation graphics positioned within plane of video stream by surgeon and aligned with point of interest; ¶126: augmentation graphics may be a surgical tool or apparatus, where computer subsystem can animate movement of a displayed patient mode, tool, or surgical apparatus to illustrate a planned procedure relative to a defined surgical site or target location on patient’s body; Note ¶14: tool attached to robotic arm; ¶139: an augmented reality navigation system is used in conjunction with a pointer tool in order to define a trajectory that a robotic surgical system can follow; ¶149: output from the augmented reality navigation system causes the robotic surgical system to automatically align with the represented trajectory and/or automatically move along the trajectory; ¶154: trajectory determined and updated based on position or orientation of pointer tool, which is used to select a plurality of points in space that are then used to collectively determine a trajectory, in response to user input signal received by computer subsystem, and trajectory is output to robotic surgical system for performing surgical procedure; ¶155: output from the augmented reality navigation system causes the robotic surgical system to automatically align with the represented trajectory and/or automatically move along the trajectory, where trajectory can be overlayed graphic on display screen)
Regarding claim 20, Johnson further discloses:
Wherein the input is received through a handheld input device that is separate from and communicatively coupled with the first portable devices (Johnson, ¶178: input provided by any kind of input device, such as mouse, keyboard)
Johnson modified by Anderson further disclose:
Wherein the input is received through a handheld input device that is separate from and communicatively coupled with the first portable devices (Anderson, ¶26: hand tracking using controller held by learner; ¶¶37-38: AR/VR display allowing input via a mechanical controller device such as a VR controller)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, and further including the use of a different type of input device for providing user input as provided by Anderson, using known electronic interfacing and programming techniques. The modification merely substitutes one known AR input device for another, yielding predictable results of providing a handheld input device for controlling HMD input in AR. The modification also results in an improved collaborative augmented reality surgical environment by allowing more specific input controls for easier implementation and accuracy of input.
Regarding claim 21, Johnson modified by Anderson further discloses:
Wherein the first portable electronic device is a virtual reality headset (Johnson, Fig. 1 and ¶22: head mounted display including semi opaque images for presenting virtual augmented graphics to user)
Johnson modified by Anderson further discloses:
Wherein the first portable electronic device is a virtual reality headset (Anderson, ¶20: head mounted VR display)
Both Johnson and Anderson are directed to virtual systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, and further including the use of a VR display of Anderson, using known electronic interfacing and programming techniques. The modification merely substitutes one known AR display for another, yielding predictable results of providing an interactive environment based on user perspective. The modification furthermore would be obvious to try as selecting from a finite number of identified predictable solutions for providing an interactive user perspective environment by using virtual reality display in place of another display type, with a reasonable expectation of success. The modification also results in an improved collaborative digital surgical environment by allowing a fully immersive environment to remove outside distractions and better focus on targeted tasks focused on by the system.
Claim(s) 6-7, 11 and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Johnson et al. (US 2019/0254754 A1) in view of
Anderson et al. (US 2021/0072947 A1) and in further view of
Maciocci et al. (US 2012/0249416 A1)
Regarding claim 6, the limitations included from claim 2 are rejected based on the same rationale as claim 2 set forth above. Further regarding claim 6, Johnson further discloses:
Wherein the first camera and the second camera (Johnson, (Johnson, ¶91; auxiliary detector in surgical environment in addition to detector disposed on HMD; ¶92: second HMD with detector) and RGBD cameras (Johnson, ¶119: sensor 602 may be RGB-D video)
Johnson modified by Anderson is not explicitly clear that multiple cameras in an AR system environment are RGB-D. This would have been obvious to use, however, at the effective filing date of the claimed invention as RGB-D cameras were conventional and well known at the time for use with AR systems, in place of other types of cameras.
Maciocci discloses:
Wherein the first camera and the second cameras are RGBD cameras (Maciocci, ¶¶113-115: HMD having RGB-D camera aligned with view of user and HMD and provide full capture and 3D reconstruction of objects within image; Fig. 2 and ¶108 further disclosing the coordinating of multiple head mounted devices in an environment and coordinating 3D map of objects and surfaces in vicinity using captured data)
Johnson, Anderson and Maciocci are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, using the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, by further utilizing the camera types of Maciocci, using known electronic interfacing and programming techniques. The modification merely substitutes one known sensor for augmented reality processing for another, yielding predictable results of using RGB-D cameras for obtaining data for use in an augmented reality system. The modification also results in an improved augmented reality surgical processing system by using more accurate sensor detectors to determine both image data and depth for more accurate augmented reality alignment of tracked data and virtual effects.
Regarding claim 7, the limitations included from claim 2 are rejected based on the same rationale as claim 2 set forth above. Further regarding claim 7, Johnson modified by Anderson further discloses:
Wherein displaying the virtual representation of the remote user comprises displaying the virtual representation of the remote user superimposed over at least a portion of the second video stream (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶22: when head-mounted AR/VR display 150 is in AR mode, graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
The only limitation not explicitly disclosed by Johnson and Anderson is the portable electronic device is a tablet computer.
Maccioci discloses:
wherein the portable electronic device is a table computer, wherein the instructions to display comprises instructions to display the virtual representation of the remote user superimposed over at least a portion of the second video stream (Maccioci, ¶3: “The present application relates to an augmented or virtual reality system using a head mounted display, or other mobile devices such as smartphones or tablets, that can place a virtual object or interface on a selected physical surface so that a single user or multiple users can collaborate to, view and interact with the virtual object on the physical surface;” ¶57: the head mounted device may comprise a processor, a memory, a display and a camera. and head mounted device 10 may be a mobile device (e.g., smartphone, etc.) that includes one or more sensors (e.g., a depth sensor, camera, etc.) for scanning or collecting information from an environment (e.g., room, etc.) and circuitry for transmitting the collected information to another device (e.g., server, second mobile device, etc.); ¶66: the head mounted display completely blocks the user's view of the room, where the head mounted display provides a virtual or augmented reality experience, where virtual objects may be added to the images projected on head mounted display, thereby appearing as real as the actual images obtained by the cameras; also ¶67: images generated by the HMD; ¶¶94-95: two users not in same room collaborate, with data sent from one HMD to another, each generating virtual objects for HMD user to see; ¶147: the processor of the first or second head mounted device may determine a visual orientation of the second user, such as by using methods described herein based on sensor data and obtained images; ¶188: rendering hands of multiple users over image)
Johnson, Anderson and Maciocci are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, using the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, by further utilizing the computer architecture for processing as provided by Maciocci, using known electronic interfacing and programming techniques. The modification merely substitutes one known computer for processing for another, yielding predictable results of using processors of a tablet device for generating the collaborative augmented reality system as opposed to other known computer types. Moreover, the modification results in an improved collaborative augmented reality surgical processing system by allowing a commonly used computer device for easier implementation and less costly implementation (i.e. using already pre-existing generic computer devices).
Regarding claim 11, the limitations included from claim 10 are rejected based on the same rationale as claim 10 set forth above. Further regarding claim 11, Johnson modified by Anderson further discloses:
wherein the at least one processor and the memory are integrated with the first portable electronic device such that the first portable electronic device generates the virtual OR (Johnson, ¶130: processors residing in HMD; ¶139: an augmented reality navigation system displays augmentation graphics that show portions of a surgical tool, surgical apparatus, and/or robotic surgical system otherwise hidden from the natural field of view of a surgeon; ¶142: unified coordinate system results from registration of AR navigation system and surgical environment, including patient, tool and robotic surgical system – i.e. virtual OR; also ¶147: unified coordinate system resulting from registration of augmented reality navigation system and surgical environment, including objects in surgical environment including patient and robotic surgical system)
Johnson further discloses:
the first portable electronic device generates the virtual room, receives the virtual representation of the remote user, and displays the virtual representation of the remote user (Anderson, ¶20: transmission of 3D space information 115 to learner room 120 enables learner 102 to see a synchronous avatar 101A of instructor 101; ¶22 discloses overlaying graphics on user’s view of physical world using 3D shared virtual space – i.e. virtual room; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space; ¶42: avatar of remote user in AR view)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Johnson modified by Anderson is not clear on the processer integrated with the first portable electronic device (i.e. the processing of the plurality of elements performed within e.g. the HMD itself). Integrating the processing components on the portable device itself, however, was well known as of the effective filing date of the claimed invention.
Maciocci discloses:
wherein the at least one processor and the memory are integrated with the first portable electronic device such that the first portable electronic device generates the virtual room, receives the virtual representation of the remote user, and displays the virtual representation of the remote user (Maccioci, ¶57: the head mounted device may comprise a processor, a memory, a display and a camera. and head mounted device 10 may be a mobile device (e.g., smartphone, etc.) that includes one or more sensors (e.g., a depth sensor, camera, etc.) for scanning or collecting information from an environment (e.g., room, etc.) and circuitry for transmitting the collected information to another device (e.g., server, second mobile device, etc.); ¶66: the head mounted display completely blocks the user's view of the room, where the head mounted display provides a virtual or augmented reality experience, where virtual objects may be added to the images projected on head mounted display, thereby appearing as real as the actual images obtained by the cameras; also ¶67: images generated by the HMD; ¶¶94-95: two users not in same room collaborate, with data sent from one HMD to another, each generating virtual objects for HMD user to see; ¶147: the processor of the first or second head mounted device may determine a visual orientation of the second user, such as by using methods described herein based on sensor data and obtained images; ¶188: rendering hands of multiple users over image)
Note that Maciocci teaches the localization of the processing of augmented reality environment data on a portable device which receives data from other devices, and that the combination of references teaches the limitations as a whole.
Johnson, Anderson and Maciocci are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, using the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, by further utilizing the computer architecture for processing as provided by Maciocci, using known electronic interfacing and programming techniques. The modification merely substitutes one known technique for distributed system processing for another, namely using localized AR device processing to perform the operations rather than client-server processing, yielding predictable results of using processors of a head-mound display device for generating the collaborative augmented reality system. Moreover, the modification results in an improved collaborative augmented reality surgical processing system by allowing for localized system processing that provides for more easily portable devices and localization of processing that allows for easier networked implementation.
Regarding claim 13, the limitations included from claim 10 are rejected based on the same rationale as claim 10 set forth above. Further regarding claim 13, Johnson modified by Anderson further discloses:
wherein the instructions to display comprises instructions to display the virtual representation of the remote user superimposed over at least a portion of the first video stream (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶22: when head-mounted AR/VR display 150 is in AR mode, graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
The only limitation not explicitly disclosed by Johnson and Anderson is the first portable electronic device is a tablet computer.
Maccioci discloses:
wherein the first portable electronic device is a table computer, wherein the instructions to display comprises instructions to display the virtual representation of the remote user superimposed over at least a portion of the first video stream (Maccioci, ¶3: “The present application relates to an augmented or virtual reality system using a head mounted display, or other mobile devices such as smartphones or tablets, that can place a virtual object or interface on a selected physical surface so that a single user or multiple users can collaborate to, view and interact with the virtual object on the physical surface;” ¶57: the head mounted device may comprise a processor, a memory, a display and a camera. and head mounted device 10 may be a mobile device (e.g., smartphone, etc.) that includes one or more sensors (e.g., a depth sensor, camera, etc.) for scanning or collecting information from an environment (e.g., room, etc.) and circuitry for transmitting the collected information to another device (e.g., server, second mobile device, etc.); ¶66: the head mounted display completely blocks the user's view of the room, where the head mounted display provides a virtual or augmented reality experience, where virtual objects may be added to the images projected on head mounted display, thereby appearing as real as the actual images obtained by the cameras; also ¶67: images generated by the HMD; ¶¶94-95: two users not in same room collaborate, with data sent from one HMD to another, each generating virtual objects for HMD user to see; ¶147: the processor of the first or second head mounted device may determine a visual orientation of the second user, such as by using methods described herein based on sensor data and obtained images; ¶188: rendering hands of multiple users over image)
Johnson, Anderson and Maciocci are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, using the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, by further utilizing the computer architecture for processing as provided by Maciocci, using known electronic interfacing and programming techniques. The modification merely substitutes one known computer for processing for another, yielding predictable results of using processors of a tablet device for generating the collaborative augmented reality system as opposed to other known computer types. Moreover, the modification results in an improved collaborative augmented reality surgical processing system by allowing a commonly used computer device for easier implementation and less costly implementation (i.e. using already pre-existing generic computer devices).
Regarding claim 14, Johnson further discloses:
wherein the virtual representation of the remote user is displayed on the screen of the first portable electronic device, while the position within the OR is captured within the first video stream (Johnson, ¶91; auxiliary detector in surgical environment in addition to detector disposed on HMD; ¶92: second HMD with detector; ¶¶124-125: real-time feeds of augmented reality of surgical location, anatomy, tool, apparatus, with surgical overlay of AR graphics on HMD; ¶127: real-time video captured by an HMD-mounted camera during surgery of a patient may be processed by computer subsystem 820 and compared to video captured by one or more other sources, e.g., an auxiliary detector of imaging equipment 830; ¶131: at least one detector tracks real-world features identifying or attached to patient and robotic surgical system, where the at least one detector 902 includes a plurality of cameras that are spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked)
Johnson modified by Anderson further discloses:
wherein the virtual representation of the remote user is displayed on the screen of the first portable electronic device (Anderson, ¶20 discloses shared 3D virtual space between learner/user and instructor, with transmission of 3D space information to learner room 120 enabling learner to see synchronous avatar 101A of instructor, indicating to learner 102 where instructor 101 is located in shared 3D virtual space 103 and where instructor 101 is looking and/or pointing in real-time; ¶22: when head-mounted AR/VR display 150 is in AR mode, graphics are overlaid on a video stream of the physical world from the current point of view of the user wearing the head-mounted AR/VR display 150; ¶27: AR and VR content based on position information for a position-tracked object in instructor room 110, such as a body part of instructor 101 and/or an object held by instructor 101; Fig. 1 and ¶33 and ¶45, both devices at remote locations allow users to see synchronous avatar and annotations in shared space)
Both Johnson and Anderson are directed to augmented reality systems that allow coordinated spatial data for presenting virtual objects in a collaborative augmented reality environment. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the collaborative augmented reality operating room system and technique coordinating augmented reality effects between users in the distributed augmented reality system as provide by Johnson, by including the coordination of remote users as virtual objects for incorporation into the augmented reality space as provided by Anderson, using known electronic interfacing and programming techniques. The modification results in an improved collaborative augmented reality surgical environment by allowing multiple users to interact with one another in a distributed system instead of limiting to local use (see e.g. Anderson, ¶9: At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a bi-directional interface that provides a mixed-reality presence for both an instructor residing in a teaching space or location and a learner residing in a learning space or location that is different than the teaching space or location.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616
Prosecution Insights