DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
The examiner notes this is a second non-final action due to the 101 rejection not previously being presented.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Regarding claim 1, it recite(s) A method comprising:
receiving one or more inputs from a user interacting with an extended reality hardware computing device including a display that is displaying an extended reality environment including a computer-generated virtual object and real-world imagery, wherein the one or more inputs provide an inquiry for additional information about the virtual object displayed in the extended reality environment to the user, wherein the one or more inputs further include the user indicating the virtual object, and the additional information is not already present at such time and location in the extended reality environment when the user provides the inquiry; retrieving a response to the inquiry; and automatically providing an electronic response as an answer to the inquiry, wherein said retrieving a response includes submitting a request for the additional information based on the inquiry being provided as at least one of text data, audio data, image data, scanned data, and object data, to an artificial intelligence process that obtains the additional information and then transmits the response that includes the additional information about the virtual object.
MPEP 2106 III provide a flowchart for the subject matter eligibility test for product and processes. The analysis following the flowchart is as follows:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Yes. It recites a method, which is a process.
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or nature phenomenon?
Yes. The claim recites an abstract idea.
retrieving a response to the inquiry;
automatically providing an electronic response as an answer to the inquiry;
retrieving a response to the inquiry is merely a mental process of thinking about a question or visually in the human mind or with assistance of a pen and paper, as retrieving the response can just be considered a human processing mentally a response which they plan to provide.
automatically providing an electronic response as an answer to the inquiry can be considered a mental process of providing the mentally determined response with the assistance of a pen and paper as there is no limit as to how the response is provided.
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
This judicial exception is not integrated into a practical application. The limitations “receiving one or more inputs from a user interacting with an extended reality hardware computing device including a display” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering and outputting. See MPEP 2106.05.
In particular, the claim only recites the additional elements an extended reality hardware computing device including a display. The extended reality hardware computing device including a display are recited at a high-level of generality (i.e., computing device with display) such that it amounts no more than mere instructions to apply the exception using a generic computer components. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
As discussed in Step 2A, Prong Two above, the recitations of “receiving one or more inputs from a user interacting with an extended reality hardware computing device including a display” are recited at a high level of generality. These elements amount to receiving data and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II.
These elements are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. In the instant claim, the use of an extended reality hardware computing device including a display presents the idea of a solution while failing to describe how the extended reality hardware computing device including a display are used or structured to achieve the solution. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Regarding claim 2, it recite(s) The method of claim 1, wherein the one or more inputs include one or more of the user speaking identification of, making reference to, holding, touching, or pointing to the virtual object or real-world object, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Regarding claim 3, it recite(s) The method of claim 1, wherein the one or more inputs include the user touching or pointing at a virtual page that includes text, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Regarding claim 4, it recite(s) The method of claim 1, wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video, which merely further defines the additional information utilized in the abstract idea of retrieving a response to the inquiry; automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 5, it recite(s) The method of claim 2, wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video, which merely further defines the additional information utilized in the abstract idea of retrieving a response to the inquiry; automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 6, it recite(s) The method of claim 2, wherein the input includes the user speaking a pronoun to reference the virtual object or real-world object, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Regarding claim 7, it recite(s) The method of claim 3, wherein the input includes the user touching or pointing at a virtual page that includes text while the user is speaking the inquiry, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Regarding claim 8, it recite(s) The method of claim 1, wherein the response includes written or spoken text, which merely further defines the information utilized in the abstract idea of automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 9, it recite(s) The method of claim 2, wherein the response includes written or spoken text, which merely further defines the information utilized in the abstract idea of automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 10, it recite(s) The method of claim 1, wherein the response audibly provides the additional information to the user, which merely further defines the information provided in the abstract idea of automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 11, it recite(s) The method of claim 2, wherein the response audibly provides the additional information to the user, which merely further defines the information provided in the abstract idea of automatically providing an electronic response as an answer to the inquiry. A narrow abstract idea is still an abstract idea and therefore it does not amount to significantly more.
Regarding claim 12, it recite(s) The method of claim 1, wherein the input includes an image captured from the real world of the user from a camera, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05. These elements amount to receiving data and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II. The additional element is recited at a high level of generality and does not amount to significantly more.
Regarding claim 13, it recite(s) The method of claim 12, further comprising receiving the image from a camera of the extended reality hardware device, amounts to mere data gathering of receiving data without significantly more. These elements amount to receiving data and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II. The additional element is recited at a high level of generality and does not amount to significantly more.
Regarding claim 14, it recite(s) The method of claim 13, wherein the camera of the extended reality hardware device is worn by the user, These elements amount to receiving data and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II. The additional element is recited at a high level of generality and does not amount to significantly more.
Regarding claim 15, it recite(s) The method of claim 14, wherein the image is received by the extended reality hardware device while in a passthrough mode that allows the user to see objects in the user's real world, the claim still amounts to receiving data which is considered a data gathering activity and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II.
Regarding claim 16, it recite(s) The method of claim 15, further comprising the extended reality hardware device receiving the image while the camera is restricted from capturing surrounding real-world environment other than subject matter of the image permitted to be received into the extended reality environment, the claim still amounts to receiving data which is considered a data gathering activity and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II.
Regarding claim 17, it recite(s) The method of claim 12, further comprising receiving the image from an external camera of a smartphone or webcam device, These elements amount to receiving data and are well understood, routine, conventional activity. See MPEP 2106.05(d), subsection II. The additional element is recited at a high level of generality and does not amount to significantly more.
Regarding claim 18, it recite(s) The method of claim 1, further comprising assessing context of the inquiry in retrieving the additional information.
MPEP 2106 III provide a flowchart for the subject matter eligibility test for product and processes. The analysis following the flowchart is as follows:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Yes. It recites a method, which is a process.
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or nature phenomenon?
Yes. The claim recites an abstract idea.
Assessing context of the inquiry;
Assessing context of the inquiry is merely a mental process of thinking about a question or visually in the human mind or with assistance of a pen and paper.
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
There are no additional elements in the claim.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
There are no additional elements in the claim.
Regarding claim 19, it recite(s) The method of claim 2, further comprising assessing context of the inquiry in retrieving the additional information.
MPEP 2106 III provide a flowchart for the subject matter eligibility test for product and processes. The analysis following the flowchart is as follows:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Yes. It recites a method, which is a process.
Step 2A, Prong One: Does the claim recite an abstract idea, law of nature, or nature phenomenon?
Yes. The claim recites an abstract idea.
Assessing context of the inquiry;
Assessing context of the inquiry is merely a mental process of thinking about a question or visually in the human mind or with assistance of a pen and paper.
Step 2A, Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No.
There are no additional elements in the claim.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No.
There are no additional elements in the claim.
Regarding claim 20, it recite(s) The method of claim 1, wherein the input includes the user gesturing to reference the virtual object or real world object while speaking the inquiry, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Regarding claim 21, it recite(s) The method of claim 1, wherein receiving the one or more inputs comprises selecting the virtual object through a virtual tablet, which merely further describes the inputs which is the data that is being gathered. See MPEP 2106.05.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 5, 8, 9-15, 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack).
Regarding claim 1, Buch teaches A method (A method for generating and providing artificial intelligence assisted surgical guidance includes analyzing video images from surgical procedure and training a neural network to identify at least one of anatomical objects, surgical objects, and tissue manipulation in video images. See abstract) comprising:
receiving one or more inputs from a user interacting with an extended reality hardware computing device including a display that is displaying an extended reality environment, wherein the one or more inputs provides an inquiry for additional information about a real-world object displayed in the extended reality environment to the user, wherein the one or more inputs further include the user indicating the real-world object, and the additional information is not already present at such time and location in the extended reality environment when the user provides the inquiry (A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]);
retrieving a response to the inquiry (The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]); and
automatically providing an electronic response as an answer to the inquiry, wherein said retrieving a response includes submitting a request for the additional information based on the inquiry being provided as at least one of text data, audio data, image data, scanned data, and object data, to an artificial intelligence process that obtains the additional information and then transmits the response that includes the additional information about the real-world object (The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]) (A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]), but is silent to wherein the object is a virtual object.
Spivack teaches users asking questions regarding virtual objects in augmented reality (A virtual object can include one or more streams of events it generates, which can be queried by actors (human users, or other VOBs) with permission See paragraph [0322])(Communicate with an object using text and/or speech to ask questions, give commands, or perform other interactions See paragraph [0324]) (Systems, Methods and Apparatuses to Facilitate Physical and Non-Physical Interaction/Action/Reaction are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, to generate a mixed reality environment associated with a physical location. The mixed reality environment can include, a photorealistic representation of a physical environment associated with the physical location and/or a virtual world having a virtual object. The virtual object can, in one embodiment, be a portal virtual object is interacted with in the mixed reality environment to traverse into or out of another reality. A human user can be enabled to discover relevant objects in the other reality through physical exploration of the physical location and areas surrounding the physical location and/or digital exploration in the other reality environment through digital navigation via a user interface of the other reality. See abstract).
Buch and Spivack teach of presenting augmented reality information to a user and utilizing questions about objects in augmented reality and Spivack teaches that the questions can be related to a virtual object, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch with the virtual object question techniques of Spivack such that the system could be extended to include questions regarding virtual objects.
Regarding claim 2, Buch in view of Spivack teaches The method of claim 1, wherein the one or more inputs include one or more of the user speaking identification of, making reference to, holding, touching, or pointing to the virtual object or real-world object (Buch; A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 4, Buch in view of Spivack teaches The method of claim 1, wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video (Buch; A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 5, Buch in view of Spivack teaches The method of claim 2, wherein the response that includes the additional information presents one or more of a picture, graphic, three-dimensional object display and video (Buch; A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 8, Buch in view of Spivack teaches The method of claim 1, wherein the response includes written or spoken text (Buch; The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]) (Buch; A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 9, Buch in view of Spivack teaches The method of claim 2, wherein the response includes written or spoken text (Buch; The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]) (Buch; A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 10, Buch in view of Spivack teaches The method of claim 1, wherein the response audibly provides the additional information to the user (Buch;This output, namely the identified key elements of the surgical field, can then be returned to the surgeon in audiovisual from on a monitor or, in the future, a heads-up augmented reality display in real-time. This will appear as segmented overlays over the specific anatomic structures and provide probability estimates for each item to demonstrate the AOA's level of certainty (or uncertainty). See paragraph [0020]).
Regarding claim 11, Buch in view of Spivack teaches The method of claim 2, wherein the response audibly provides the additional information to the user (Buch; This output, namely the identified key elements
of the surgical field, can then be returned to the surgeon in audiovisual from on a monitor or, in the future, a heads-up augmented reality display in real-time. This will appear as segmented overlays over the specific anatomic structures and provide probability estimates for each item to demonstrate the AOA's level of certainty (or uncertainty). See paragraph [0020]).
Regarding claim 12, Buch in view of Spivack teaches The method of claim 1, wherein the input includes an image captured from the real world of the user from a camera (Buch;we foresee this technology being used on a digital, augmented reality enhanced, optical zoom capable surgical loupe headset where the video feed and subsequent visual feedback could be captured by and returned to the surgical loupe headset. As surgical headset technology improves, this type of AI system could become part of the surgical headpiece worn by every surgeon. This wide-reaching application ofreal-time, automated decisionmaking augmentation, surgical roadmaps, complication avoidance warnings, procedural stepwise cost estimates, predictive analytics, and tailored surgical guidance using artificial intelligence for surgeons is entirely novel; and our innovative AOA could redefine the forefront of cutting-edge surgical care. This technology also has the potential to transform surgical training and medical education. Trainees would gain valuable knowledge about anatomy and the operative field that most often takes years to hone. Medical students would be able to observe surgeries with the output of the AOA overlaid to improve their learning of anatomy. See paragraph [0025]).
Regarding claim 13, Buch in view of Spivack teaches The method of claim 12, further comprising receiving the image from a camera of the extended reality hardware device (Buch;we foresee this technology being used on a digital, augmented reality enhanced, optical zoom capable surgical loupe headset where the video feed and subsequent visual feedback could be captured by and returned to the surgical loupe headset. As surgical headset technology improves, this type of AI system could become part of the surgical headpiece worn by every surgeon. This wide-reaching application ofreal-time, automated decisionmaking augmentation, surgical roadmaps, complication avoidance warnings, procedural stepwise cost estimates, predictive analytics, and tailored surgical guidance using artificial intelligence for surgeons is entirely novel; and our innovative AOA could redefine the forefront of cutting-edge surgical care. This technology also has the potential to transform surgical training and medical education. Trainees would gain valuable knowledge about anatomy and the operative field that most often takes years to hone. Medical students would be able to observe surgeries with the output of the AOA overlaid to improve their learning of anatomy. See paragraph [0025]).
Regarding claim 14, Buch in view of Spivack teaches The method of claim 13, wherein the camera of the extended reality hardware device is worn by the user (Buch;we foresee this technology being used on a digital, augmented reality enhanced, optical zoom capable surgical loupe headset where the video feed and subsequent visual feedback could be captured by and returned to the surgical loupe headset. As surgical headset technology improves, this type of AI system could become part of the surgical headpiece worn by every surgeon. This wide-reaching application ofreal-time, automated decisionmaking augmentation, surgical roadmaps, complication avoidance warnings, procedural stepwise cost estimates, predictive analytics, and tailored surgical guidance using artificial intelligence for surgeons is entirely novel; and our innovative AOA could redefine the forefront of cutting-edge surgical care. This technology also has the potential to transform surgical training and medical education. Trainees would gain valuable knowledge about anatomy and the operative field that most often takes years to hone. Medical students would be able to observe surgeries with the output of the AOA overlaid to improve their learning of anatomy. See paragraph [0025]).
Regarding claim 15, Buch in view of Spivack teaches The method of claim 14, wherein the image is received by the extended reality hardware device while in a passthrough mode that allows the user to see objects in the user's real world (Buch; In the future, we foresee this technology being used on a digital, augmented reality enhanced, optical zoom capable surgical loupe headset where the video feed and subsequent visual feedback could be captured by and returned to the surgical loupe headset. As surgical headset technology improves, this type of AI system could become part of the surgical headpiece worn by every surgeon. This wide-reaching application ofreal-time, automated decisionmaking augmentation, surgical roadmaps, complication avoidance warnings, procedural stepwise cost estimates, predictive analytics, and tailored surgical guidance using artificial intelligence for surgeons is entirely novel; and our innovative AOA could redefine the forefront of cutting-edge surgical care. This technology also has the potential to transform surgical training and medical education. Trainees would gain valuable knowledge about anatomy and the operative field that most often takes years to hone. Medical students would be able to observe surgeries with the output of the AOA overlaid to improve their learning of anatomy. See paragraph [0025]).
Regarding claim 17, Buch in view of Spivack teaches the method of claim 12, further comprising receiving the image from an external camera of a smartphone or webcam device (Buch;In all operations, a video feed of the operative field can be obtained with either an overhead operating room (OR) camera, microscope, or endoscope. Regardless of the video input, the feed will be analyzed on a trained, multilayer, convolutional deep neural network capable of identifying key anatomical objects, surgical objects, and tissue manipulation. This real-time feedback and output of the trained DNN is considered the Artificial Operative Assistant (AOA). See paragraph [0018]).
Regarding claim 18, Buch in view of Spivack teaches The method of claim 1, further comprising assessing context of the inquiry in retrieving the additional information (Buch;The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]) (Buch;A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Regarding claim 19, Buch in view of Spivack teaches The method of claim 2, further comprising assessing context of the inquiry in retrieving the additional information (Buch; The AOA provides real-time automated feedback on the surgical field to augment a surgeon's decision making and has the potential to become essential to any surgical procedure, constituting a new gold standard. See paragraph [0016]) (Buch;A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. "How certain are you that this is the disc material?", "How safe is it to retract this part of the thecal sac?") or ask the AOA to label objects as the user sees fit (i.e. "Label this object as the carotid artery."). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output. See paragraph [0024]).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack) in view of Nathie (“How to Get Hand Tracking on The Oculus Meta Quest 2”, 2020,
https://www.youtube.com/watch?v=hX7GJfYmj7M)(Hereinafter referred to as Nathie).
Regarding claim 3, Buch in view of Spivack teaches the method of claim 1, but is silent to wherein the one or more inputs include the user touching or pointing at a virtual page that includes text.
Nathie teaches a head mounted display that uses allowed the user to control various touch interactions including on a virtual web page and interacting with a virtual home screen including various applications (See figure 1 in this action and 4:58)
PNG
media_image1.png
720
1282
media_image1.png
Greyscale
Figure 1: Hand Gestures Interacting with web browser and various applications
Buch in view of Spivack and Nathie teach of presenting virtual information and Nathie teaches that gestures can be used to perform intuitive interactions without controllers, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack with the hand gesture interaction techniques of Nathie such that the user would be able to interact with various interfaces without the need for an external controller.
Claim(s) 6 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack)
in view of Uchino et al. (“VR Interaction in Real-Time between Avatar with Voice and Gesture Recognition System”, IEEE, 2007)(Hereinafter referred to as Uchino).
Regarding claim 6, Buch in view of Spivack teaches the method of claim 2, but is silent to wherein the input includes the user speaking a pronoun to reference the virtual object or real-world object.
Uchino teaches a multi-modal input technique in which the user combines both a pointing gesture and a speech in conjunction to interact with a system (Avatar: Which pen do you want to buy? User: Please show me this one. (See figure 2)(The starting and ending time of each demonstrative pronoun appearing in utterance detected with Julius are used to judge the temporal relation between utterance of a demonstrative pronoun and pointing action. This function successfully finds temporal relation even if utterance includes several demonstrative pronouns, and several pointing actions are given. See section 4.1 Spoken Language Processing, first paragraph)(Pointing a red pen) (1) Avatar: Which color do you like? User: Please give the red one. (The avatar is going to reaching his hand to one of two red pens.) (2) User: No, it is wrong. Avatar: Is this one? (Grasping another red pen) (3) See page 4, right col., below figure 5 and figure 5)
Buch in view of Spivack and Uchino teach interacting computer systems with voice inputs and Uchino teaches that the user input can be multi-modal such that both a pointing gesture and speech are used to recognize the user’s intent, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack with the multimodal input and object identification techniques of Uchino such that the user could use a combination of virtual gestures and natural speech to allow the system to identify which object they were interested in.
Regarding claim 20, Buch in view of Spivack teaches the method of claim 1, but is silent to wherein the input includes the user gesturing to reference the virtual object or real world object while speaking the inquiry.
Uchino teaches a multi-modal input technique in which the user combines both a pointing gesture and a speech in conjunction to interact with a system (Avatar: Which pen do you want to buy? User: Please show me this one. (See figure 2)(The starting and ending time of each demonstrative pronoun appearing in utterance detected with Julius are used to judge the temporal relation between utterance of a demonstrative pronoun and pointing action. This function successfully finds temporal relation even if utterance includes several demonstrative pronouns, and several pointing actions are given. See section 4.1 Spoken Language Processing, first paragraph)(Pointing a red pen) (1) Avatar: Which color do you like? User: Please give the red one. (The avatar is going to reaching his hand to one of two red pens.) (2) User: No, it is wrong. Avatar: Is this one? (Grasping another red pen) (3) See page 4, right col., below figure 5 and figure 5)
Buch in view of Spivack and Uchino teach interacting computer systems with voice inputs and Uchino teaches that the user input can be multi-modal such that both a pointing gesture and speech are used to recognize the user’s intent, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack with the multimodal input and object identification techniques of Uchino such that the user could use a combination of virtual gestures and natural speech to allow the system to identify which object they were interested in.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack) in view of in view of Nathie (“How to Get Hand Tracking on The Oculus Meta Quest 2”, 2020,
https://www.youtube.com/watch?v=hX7GJfYmj7M)(Hereinafter referred to as Nathie) in view of Uchino et al. (“VR Interaction in Real-Time between Avatar with Voice and Gesture Recognition System”, IEEE, 2007)(Hereinafter referred to as Uchino).
Regarding claim 7, Buch in view of Spivack in view of Nathie teaches the method of claim 3, but is silent to wherein the input includes the user touching or pointing at a virtual page that includes text (Nathie See figure 1 in this action and 4:58)), but is silent to while the user is speaking the inquiry.
Uchino teaches a multi-modal input technique in which the user combines both a pointing gesture and a speech in conjunction to interact with a system (Avatar: Which pen do you want to buy? User: Please show me this one. (See figure 2)(The starting and ending time of each demonstrative pronoun appearing in utterance detected with Julius are used to judge the temporal relation between utterance of a demonstrative pronoun and pointing action. This function successfully finds temporal relation even if utterance includes several demonstrative pronouns, and several pointing actions are given. See section 4.1 Spoken Language Processing, first paragraph)(Pointing a red pen) (1) Avatar: Which color do you like? User: Please give the red one. (The avatar is going to reaching his hand to one of two red pens.) (2) User: No, it is wrong. Avatar: Is this one? (Grasping another red pen) (3) See page 4, right col., below figure 5 and figure 5)
Buch in view of Spivack in view of Nathie and Uchino teach interacting computer systems with voice inputs and pointing gestures and Uchino teaches that the user input can be multi-modal such that both a pointing gesture and speech are used to recognize the user’s intent, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack in view of Nathie with the multimodal input and object identification techniques of Uchino such that the user could use a combination of virtual gestures and natural speech to allow the system to identify which object they were interested in.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack) in view of Paradise Decay (“Oculus Quest Passthrough Camera & Guardian System in Action”, https://www.youtube.com/watch?v=nQQJu1WV4_A, 2019)(Hereinafter referred to as Paradise)
Regarding claim 16, Buch in view of Spivack teaches the method of claim 15, but is silent to further comprising the extended reality hardware device receiving the image while the camera is restricted from capturing surrounding real-world environment other than subject matter of the image permitted to be received into the extended reality environment.
Paradise teaches a HMD that allows for capturing the users environment using a guardian system allowing the user to set up their safe workable area (See figures 2 and 3 in this action) (Paradise; See figure 2 in this action, 1:26)
PNG
media_image2.png
552
975
media_image2.png
Greyscale
Figure 2: Setting up guardian system in pass through mode
PNG
media_image3.png
547
975
media_image3.png
Greyscale
Figure 3: guardian boundary visible in VR mode
Buch in view of Spivack and Paradise teach of presenting virtual objects to users and Paradise teaches that a pass through guardian mode can be provided to set up a boundary to avoid colliding with objects in the real environment when not in pass through mode, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack with the guardian mode techniques of Paradise such that the user could avoid colliding with objects in the real environment while in virtual reality.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al (US 2021/0307841)(Hereinafter referred to as Buch) in view of Spivack et al. (US 2019/0107935)(Hereinafter referred to as Spivack) in view of Richardson et al. (US 2019/0310688)(Hereinafter referred to as Richardson)
Regarding claim 21, Buch in view of Spivack teaches The method of claim 1, but is silent to wherein receiving the one or more inputs comprises selecting the virtual object through a virtual tablet.
Richardson teaches an augmented reality virtual tablet for interacting in augmented reality (Virtual scene 150 includes an augmented reality keyboard 152 and other virtual elements, such as virtual elements 154 and 156. Augmented reality keyboard 152 could be any virtual device, including a virtual keyboard, a virtual tablet, or another virtualized representation of a physical input device, for example and without limitation. User 140 interacts with augmented reality keyboard 152 by performing typing motions using hands 142. These typing motions generally correspond to keystrokes user 140 performs to express a text sequence. Imaging device 130 captures video data depicting these typing motions and transmits the video data to console 120.),
Buch in view of Spivack and Richardson teach of a user interacting in augmented reality and Richardson teaches that interactions can be performed on a virtual tablet, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Buch in view of Spivack with the virtual elements techniques and virtual tablet of Richardson such that a user could interact with the augmented reality space based on gestures similar to those performed in physical space.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS R WILSON whose telephone number is (571)272-0936. The examiner can normally be reached M-F 7:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (572)-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS R WILSON/Primary Examiner, Art Unit 2611