DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21, 24-26, 28, 31, 34-35, and 37-39 are rejected under 35 U.S.C. 103 as being unpatentable over Garcia et al (US 20190005848 A1) in view of Goel et al (US 20190318640 A1).
Regarding claim 21, A method for providing an immersive mobile interface for simulation and teleoperation of a surgical robotic system (Garcia [0007], “The head-mounted display may include an immersive display (an immersive mobile interface) for displaying the virtual robotic surgical environment (simulation) to the user”; [0013], “a method for facilitating visualization of a virtual robotic surgical environment”; [0042], “user console 100 to remotely manipulate the robotic arms 160 and/or surgical instruments (e.g., tele-operation)”), the method comprising:
receiving, by a head-worn device that is being worn by a user, a first video stream that comprises a virtual surgical robot within a virtual environment (Garcia [0136], “first-person perspective view of the virtual robotic surgical environment”; [0140], “the pelvic region of a virtual patient and facing the patient's abdomen so as to provide a “virtual video feed” of the patient's abdomen. Once placed, a virtual camera 1220 may be subsequently repositioned. A camera view (e.g., a circular inset view, or window of any suitable shape) may be placed on the HUD as a window view showing the virtual video feed”);
displaying, on a display of the head-worn device, a first view section that comprises the first video stream and a second view section that comprises the second video stream (Garcia [0120], “a first window view of the virtual robotic surgical environment from the second vantage point may be displayed in the first portal 910a ... a second window view of the virtual robotic surgical environment may be displayed in the second portal 910b (a first view section that comprises the first video stream and a second view section that comprises the second video stream).”).
Garcia does not disclose
receiving, by the head-worn device, a second video stream that comprises a view of a portion of the user's body;
However, Goel discloses
receiving, by the head-worn device, a second video stream that comprises a view of a portion of the user's body (Goel fig. 6; [0084], “In FIG. 6, the virtual hand 200 is moving virtual fragment 236 (interpreted as reading on a second stream that shows hands in relation to a first stream depicted as 222).”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Garcia with Goel to display a portion of a user’s hands during a robotic simulation. This would have been done to generate a more realistic view of the simulated environment. See, for example, Goel [0079], “Features such as inclusion of a realistic virtual table 208 and realistic walls and floor may add to the realism of the virtual world.”
Regarding claim 24, Garcia in view of Goel discloses the method of claim 21, wherein the portion of the user's body comprises a foot or a hand of the user (Goel fig. 6; [0084], “In FIG. 6, the virtual hand 200 is moving virtual fragment 236”).
Regarding claim 25, Garcia in view of Goel discloses the method of claim 24, wherein the second video stream further comprises a user control operable by the foot or the hand of the user, or a virtualization of the user control (Goel [0070], “The virtual hand 200 is holding a virtual medical device 202 (a user control operable by the hand of the user)”).
Regarding claim 26, Garcia in view of Goel discloses the method of claim 21 further comprising:
receiving, by the head-worn device, tracking data from a tracking system that is integrated into the head-worn device (Garcia [0055], “virtual reality processor 210 may be in communication with … head-mounted display … the virtual reality control 210 may receive input from sensors in the handheld controllers 230 to determine location and orientation of the handheld controllers 230, which may be used … to display in the head-mounted display 220 to the user (tracking data received by the HMD)”); and
manipulating, by the head-worn device and responsive to receiving the tracking data, the virtual surgical robot (Garcia [0007], “a virtual robotic surgical environment within which a user may … interact with virtual objects via the head-mounted display”; [0055], “the virtual reality control 210 may receive input from sensors in the handheld controllers 230 … to display in the head-mounted display 220 to the user, as well as translate user input (for interacting with the virtual environment) into corresponding modifications of the virtual robotic surgical environment (interpreted as reading on manipulating the surgical robot thru the HMD using the processor)”).
Claim 28 recites a head-worn device which corresponds to the function performed by the method of claim 21. As such, the mapping and rejection of claim 21 above is considered applicable to the head-worn device of claim 28.
Additionally discloses, A head-worn device (Garcia [0007], “a head-mounted display”) comprising:
a display (Garcia [0007], “The head-mounted display may include an immersive display”);
one or more processors Garcia [0007], “virtual reality processor”); and
memory having stored therein instructions which when executed by the one or more processors (Garcia [0007], “a processor in a computer implementing instructions stored in memory”).
Regarding claim 31, Garcia in view of Goel discloses the head-worn device of claim 28, wherein the video stream is a first video stream, wherein the memory has further instructions to:
receive, during the simulation or the teleoperation of the surgical robotic operating procedure and while the head-worn device is worn by the user, a second video stream that comprises the view of the foot or the hand of the user of the head-worn device (Goel [0070], “The virtual hand 200 is holding a virtual medical device 202 (a user control operable by the hand of the user)”); and
display, on the display, the rendering of the first video stream within the first view section and a rendering of the second video stream in the second view section (Garcia [0120], “a first window view of the virtual robotic surgical environment from the second vantage point may be displayed in the first portal 910a ... a second window view of the virtual robotic surgical environment may be displayed in the second portal 910b (a first view section that comprises the first video stream and a second view section that comprises the second video stream).”).
Claim 34 recites a head-worn device which corresponds to the function performed by the method of claim 25. As such, the mapping and rejection of claim 25 above is considered applicable to the head-worn device of claim 34.
Claim 35 recites a processor which corresponds to the function performed by the method of claim 21. As such, the mapping and rejection of claim 21 above is considered applicable to the processor of claim 35.
Additionally Garcia discloses a processor (Garcia [0007], “virtual reality processor”).
Regarding claim 37, Garcia in view of Goel discloses the processor of claim 35 receives the second video stream by producing the second video stream that includes the foot or the hand of the user within the view or a virtual representation of the foot or the hand of the user (Goel [0070], “The virtual hand 200 is holding a virtual medical device 202”).
Claim 38 recites a processor which corresponds to the function performed by the method of claim 25. As such, the mapping and rejection of claim 25 above is considered applicable to the processor of claim 38.
Claim 39 recites a processor which corresponds to the function performed by the method of claim 26. As such, the mapping and rejection of claim 26 above is considered applicable to the processor of claim 39.
Claims 22-23, 32-33 and 36 are rejected under 35 U.S.C. 103 as being unpatentable over Garcia in view of Goel and further view of Lang et al (US 20210192759 A1).
Regarding claim 22, Garcia in view of Goel discloses the method of claim 21, but does not disclose wherein receiving the second video stream comprises capturing, using a camera of the head-worn device, image data that includes the view of the portion of the user's body.
However, Lang discloses
receiving the second video stream comprises capturing, using a camera of the head-worn device, image data that includes the view of the portion of the user's body (Lang [0181], “head mounted displays can be used… with one or more … video capture systems … to image the live data … of the surgeon's body, e.g. his or her fingers, hands or arms.”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Garcia further with Lang to utilize a camera of an HMD to capture portions of a user’s body. This would have been done to generate an accurate representation of a surgical site in a proper viewing perspective for the user.
Regarding claim 23, Garcia in view of Goel and further view of Lang discloses the method of claim 22, wherein receiving the second video stream comprises producing the second video stream as a virtual representation of the view of the portion of the user's body using the image data (Lang discloses image data for a portion of a user’s body and Goel discloses - Goel fig. 6; [0084], “In FIG. 6, the virtual hand 200 is moving virtual fragment 236” The motivation to combine would have been the same as for claim 22.).
Claim 32 recites a head-worn device which corresponds to the function performed by the method of claim 22. As such, the mapping and rejection of claim 22 above is considered applicable to the head-worn device of claim 32.
Claim 33 recites a head-worn device which corresponds to the function performed by the method of claim 23. As such, the mapping and rejection of claim 23 above is considered applicable to the head-worn device of claim 33.
Regarding claim 36, Garcia in view of Goel discloses the processor of claim 35, but does not disclose wherein the second video stream comprises image data captured by a camera of the head-worn device that includes a view of the foot or the hand of the user, while the head-worn device is worn on a head of the user.
However, Lang discloses
wherein the second video stream comprises image data captured by a camera of the head-worn device that includes a view of the foot or the hand of the user, while the head-worn device is worn on a head of the user (Lang [0181], “head mounted displays can be used… with one or more … video capture systems … to image the live data … of the surgeon's body, e.g. his or her fingers, hands or arms.”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Garcia further with Lang to utilize a camera of an HMD to capture portions of a user’s body. This would have been done to generate an accurate representation of a surgical site in a proper viewing perspective for the user.
Claims 27, 30 and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Garcia in view of Goel and further view of Tako et al (US 20170367771 A1).
Regarding claim 27, Garcia in view of Goel discloses the method of claim 21, but does not disclose wherein the first view section is larger than and is positioned above the second view section within the display.
However, Tako discloses
the first view section is larger than and is positioned above the second view section within the display (Tako fig. 4A – exemplary first view 300 is larger than and positioned above a second view that display controls).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Garcia further with Tako to display a larger main view above a smaller controller related view. This would have enabled users to clearly view critical areas while having access to controller related views as well.
Claim 30 recites a head-worn device which corresponds to the function performed by the method of claim 27. As such, the mapping and rejection of claim 27 above is considered applicable to the head-worn device of claim 30. (Tako fig. 4A – exemplary first view 300 is larger than and positioned above a second view that display controls comprising controls for hand based inputs).
Claim 40 recites a processor which corresponds to the function performed by the method of claim 27. As such, the mapping and rejection of claim 27 above is considered applicable to the processor of claim 40.
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Garcia in view of Goel and further view of Barber et al (US 20210233429 A1).
Regarding claim 29, Garcia in view of Goel discloses the head-worn device of claim 28, but does not disclose wherein the second view section comprises an unobstructed opening of the display.
However, Barber discloses
the second view section comprises an unobstructed opening of the display (Barber fig. 5B; [0055], “view the rendered video image by aiming … virtual reality goggles 140, at the virtual operating room monitor 502 (a first view section) … Also visible in the VR goggle display may be … endoscope 512, and surgical tool 514 (second view section comprises an unobstructed opening of the display)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Garcia further with Barber to provide users with an unobstructed views of an operating environment. This would have allowed users to interact with the environment while viewing a virtual representation in a different view.
Conclusion
See the notice of references cited (PTO-892) for prior art made of record, including art that is not relied upon but considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JITESH PATEL whose telephone number is (571)270-3313. The examiner can normally be reached 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A. Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JITESH PATEL/Primary Examiner, Art Unit 2612