DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 were previously pending and subject to a non-final office action filed on November March 3, 2026 (the “March 3, 2026 Non-Final Office Action”). Following the March 3, 2026 Non-Final Office Action, Applicant amended claims 1, 8, and 14, in an amendment filed on June 2, 2026 (the “June 2, 2026 Amendment”), see Applicant’s amended claims (pp. 2-6 of the June 2, 2026 Amendment). As such, claims 1-20, as recited in the June 2, 2026 Amendment, are currently pending and subject to the final office action below.
Response to Applicant’s Remarks
Response to Applicant’s Remarks Concerning Rejections under 35 U.S.C. § 112(b)
Applicant’s arguments, see Applicant’s Remarks, p. 7, Rejections under 35 U.S.C. § 112 Section, filed June 2, 2026, with respect to the rejections of claims 1-7 under 35 U.S.C. § 112(b) have been considered, but they are moot in light of Applicant’s amendments to the claims. Specifically, Applicant amended the claims to overcome the indefiniteness issue identified in claim 1 in the March 3, 2026 Non-Final Office Action. Therefore, the indefinite rejections of claims 1-7 under § 112(b) are no longer necessary and are withdrawn.
Response to Applicant’s Remarks Concerning Rejections under 35 U.S.C. § 102
Applicant’s arguments, see Applicant’s Remarks, pp. 7-8, Rejections under 35 U.S.C. § 102 Section, filed June 2, 2026, with respect to rejections of claim 1, 2, 4-8, 10-15, and 17-19 under 35 U.S.C. § 102, have been fully considered, but they are moot in light of Applicant’s amendments to independent claims 1, 8, and 14. Examiner notes that Applicant’s arguments for withdrawing the § 102 rejections are solely based on the new limitations added in the June 2, 2026 Amendment. Therefore, the combination of the references previously cited in the March 3, 2026 Non-Final Office Action, are not relied upon to teach the newly amended claim limitations in claims 1, 8, and 14. Consequently, any arguments pertaining to the newly amended claim limitations are moot. Please see the amended rejections under the Claim Rejections – 35 U.S.C. § 103 Section below, for further clarification and complete analysis.
Response to Applicant’s Remarks Concerning Rejections under 35 U.S.C. § 103
Applicant’s arguments, see Applicant’s Remarks, pp. 8 and 9, Rejections under 35 U.S.C. § 103 Section, filed June 2, 2026, with respect to rejections of claim 3, 9, 16, and 20 under 35 U.S.C. § 103, have been fully considered, but they are moot in light of Applicant’s amendments to independent claims 1, 8, and 14. Examiner notes that Applicant’s arguments for withdrawing the § 103 rejections are solely based on the new limitations added in the June 2, 2026 Amendment. Therefore, the combination of the references previously cited in the March 3, 2026 Non-Final Office Action, are not relied upon to teach the newly amended claim limitations in claims 1, 8, and 14. Consequently, any arguments pertaining to the newly amended claim limitations are moot. Please see the amended rejections under the Claim Rejections – 35 U.S.C. § 103 Section below, for further clarification and complete analysis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4-8, 10-15, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over:
- Shakil et al. (Pub. No. US 2014/0222526), in view of:
- Curl (Pub. No. WO 2009/137688).
Regarding claim 1,
- Shakil et al. (Pub. No. US 2014/0222526) discloses:
- a healthcare delivery system comprising (Shakil, paragraph [0015]; Paragraph [0015] discloses an embodiment of a system 100 for augmenting performance of a provider.):
- an augmented reality device including a camera configured to capture data corresponding to a medical procedure (Shakil, paragraphs [0018], [0019], [0024], and [0025]; Paragraph [0024] discloses that the computer device 600 can be a wearable head-mounted computing device 602, which can be the VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., an augmented reality device which includes a camera). Paragraph [0025] discloses that the video camera 620 can be configured to capture images and may be forward facing to capture at least a portion of the real-world view perceived by the user to generate an augmented reality where computer generated images appear to interact with the real-world view perceived by the user (i.e., capturing data). Paragraphs [0018] and [0019] disclose that these interactions may be interactions between a medical provider and a patient, including conversations between the provider and the patient, wherein the patient provides symptoms, progress, concerns, medication information, allergy information, insurance information, and/or any other suitable health-related information to the provider; transactions wherein the patient provides demographic and/or family history information to the provider; interactions wherein the provider facilitates performance or acquisition of lab tests for the patient; interactions wherein the provider generates image data (e.g., from x-rays, MRIs, CT scanning, ultrasound scanning, etc.) from the patient; interactions wherein the provider generates other health metric data (e.g., cardiology-related data, respiratory data) from the patient; and/or any other suitable interaction between the provider and the patient (see Shakil, paragraph [0018]); an interactive session wherein the provider is examining the patient in a clinical setting or in the examining room of an office or other healthcare facility and eliciting information from the patient by questioning the patient; interactions in a hospital emergency room; interactions in an operating suite where the patient is unconscious; and interactions in a patient’s home, research setting, etc. (see Shakil, paragraph [0019]) (i.e., examples showing that the captured medical multimedia data is medical multimedia in connection with a medical procedure).);
- a network connection of the augmented reality device with a computing device over a network (Shakil, paragraph [0021]; Paragraph [0021] discloses that the computing device 600 preferably enables transmission of data generated using the computing device 600 by way of a communication link 410 (e.g., a wired connection, a wireless connection) that can be configured to communicate with a remote device (i.e., a network connection of the augmented reality device with a computing device over a network).), the computing device configured to generate feedback data for at least a portion of the data, captured by the camera, using one or more user input devices associated with the computing device (Shakil, paragraphs [0031] and [0054]; Paragraph [0031] discloses that the scribe cockpit 120 (i.e., an example of a remote device described in paragraph [0021] of Shakil, where the remote device is interpreted to be the equivalent of Applicant’s computing device) enables a scribe to receive information from interactions between a patient and the provider, which can be used to provide guidance and/or feedback to the provider (i.e., generating feedback data for at least a portion of the data captured by the camera using one or more user input devices associated with the computing device). Further, paragraph [0054] discloses that the set of tools [which the scribe is provided] include providing (1) options (e.g., by drop-down menus, by auto-completing partially inputted information) and (2) audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) (i.e., one or more user input devices associated with the computing device) to facilitate information retrieval for template completion and multimedia capture and incorporation of multimedia into content generated or prepared by the scribe and enables the scribe to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) (i.e., generating feedback data for a portion of the data captured by the camera using one or more user input devices associated with the computing device) to improve performance.), …; and
- a display of the augmented reality device configured to present the feedback data incorporated with the portion of the data superimposed over a real-world view at the display (Shakil, paragraphs [0024] and [0061], Paragraph [0061] discloses that the feedback can be received at an embodiment of the provider workstation and/or the mobile provider interface (i.e., presenting the feedback from the third party device on the display of the provider’s augmented reality device). Paragraph [0024] discloses that combining displaying capabilities and transparency can facilitate an augmented reality or heads-up display wherein a projected image or graphic is superimposed over a real-world view as perceived by the user through the lens elements 610, 612 (i.e., the feedback data that is received from third party devices may be displayed on the transparent lens of the provider’s augmented reality device and super-imposed over the real-world view of the provider during patient encounters).), the feedback data and the portion of the data displayed simultaneously on the display (Shakil, paragraph [0029], [0054], [0060], and [0061]; Paragraph [0029] discloses that the computing device 600 (i.e., augmented reality device) with the mobile provider interface 110 allows a provider to summon information from one or more sources, and to receive a response (e.g., at the computing device). The sources can be electronic databases, scheduling systems and tools, electronic information sources (e.g., Wikipedia, PUBMED, UPTODATE, EPOCRATES), and electronic health records (i.e., medical multimedia data), can be mediated by a scribe operating at a scribe cockpit 120. In variations, the response can be provided and/or rendered at a display of a computing device 600 accessible by the provider during interactions with the patient (i.e., displaying the medical multimedia data on the display of the provider’s augmented reality device), and/or during review of content generated by the scribe (i.e., displaying feedback data that is transmitted from the third party device). Further, paragraph [0054] discloses that the scribe tools can also enable a scribe (i.e., a third party) to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) to improve performance. Paragraph [0060] discloses that the feedback to the scribe and/or another entity (e.g., regarding quality of content generated by the scribe) in a qualitative (e.g., free form text/verbal feedback) and/or quantitative (e.g., using a scale of values) manner. Paragraph [0061] discloses that the feedback can be received at an embodiment of the provider workstation and/or the mobile provider interface (i.e., where paragraph [0015] discloses that the mobile provide interface is coupled to the display of the augmented reality device worn by the provider). Since the feedback can be provided to the provider’s computing device in real time in the form of free text or scales of values, this disclosure is interpreted as being the equivalent of providing the feedback data simultaneously with the other health information from various source described in paragraph [0029] (i.e., the medical multimedia data) (also see paragraph [0015] and Figure 1 where the mobile provider interface 110 is coupled to the display 112 worn by the provider). Therefore, Shakil explicitly discloses that the feedback data is displayed simultaneously on the provider’s display on the provider’s goggles, in real-time, during the medical encounter with the health information from the various sources in paragraph [0029].).
- Shakil does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Curl (Pub. No. WO 2009/137688) teaches a system, wherein:
- the feedback data includes a moving cursor, within an image or a video viewable on the augmented reality device, to indicate a portion of the image or the video (Curl, paragraphs [0042], [0073], and [0074]; Paragraph [0074] teaches that a senior or first physician can be located in a control room or remote location while a second physician, e.g., another physician, fellow or Physician’s Assistant, carries out an invasive procedure in an operating room or procedure room. The first physician can monitor the procedure and communicate with the second physician via audio and graphical mode (i.e., provide feedback), e.g., voice communication over the audio communication subsystem and annotations which are displayed on the main display device 120. The first physician can point to and identify specific items, e.g., features of anatomy, data displayed from various monitoring equipment, vital signs, etc., which are displayed on the main display 120 (i.e., pointing to and identifying specific items on the display control room which are displayed on the main display naturally the feedback includes “moving a cursor to indicate a portion of the image or the video”). The first physician can make the annotations on an auxiliary display 205 located in the control room or remote location, yet these markings will be simultaneously displayed in the operating room (i.e., the feedback data is within an image or a video viewable on the augmented reality device). Paragraph [0042] also explicitly teaches that the integration system 100 provides a cursor on the main high-resolution video display 120 which can be moved and altered using the touchpad 242, keyboard 202, and/or mouse controller 204 located in the control room, and this can allow a control-room participant to draw the attention of an operating-room participant to particular data displayed on the main high- resolution video display 120 (i.e., moving a cursor to indicate a portion of the image or the video). Paragraphs [0073] and [0074] teach that this feature is beneficial for pointing or drawing attention to particular items on a display device and is useful for providing assistance, instructional, oversight, clinical review, or analytical purposes.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to modify the system for augmenting performance of a provider taught by Shakil, to incorporate a step and feature directed to providing feedback data that includes a moving cursor within an image or video viewable on the augmented reality device to indicate a portion of the image or the video, as taught by Curl, in order to point to or draw attention to particular items on a display device and provide assistance, instructions, oversight, clinical review, or analytical purposes. See Curl, paragraph [0073] and [0074]; see also MPEP § 2143 G.
Regarding claim 2,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 2 depends on), as described above.
- Shakil further teaches a system, wherein:
- the augmented reality device includes a wearable headset device (Shakil, paragraph [0024]; Paragraph [0024] teaches that the computer device 600 can be a wearable head-mounted computing device 602 (i.e., the augmented reality device includes a wearable headset device), which can be the VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., examples of wearable headset devices).).
The motivation and rationale for modifying the system for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 1 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 4,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 4 depends on), as described above.
- Shakil further teaches a system, wherein:
- the augmented reality device includes a microphone (Shakil, Abstract and paragraph [0020]; Paragraph [0020] teaches that the computing device 600 includes an audio sensor, where the Abstract discloses that the computing device is a microphone.), and
- the data includes medical multimedia data generated with both the camera and the microphone (Shakil, paragraph [0032]; Paragraph [0032] teaches scribe cockpit interface 122 preferably couples to a display and a speaker, in order to transmit video and audio streams from provider-patient interactions (i.e., the data includes medical multimedia data generated with both the camera and the microphone).).
The motivation and rationale for modifying the system for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 1 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 5,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 5 depends on), as described above.
- Shakil further teaches a system, wherein:
- the data includes medical multimedia data manipulated in space responsive to one or more gestures detected by the camera (Shakil, paragraph [0054]; Paragraph [0054] teaches that the system includes a set of tools which provide audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) (i.e., one or more gestures that are detected by the camera) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) (i.e., the data includes medical multimedia data that is manipulated in space responsive to one or more gestures) to facilitate information retrieval for template completion. Paragraph [0054] teaches that providing audio and/or video manipulation tools can facilitate multimedia capture and incorporation of multimedia (e.g., selected image/video clips, edited image/videos) into content (i.e., the one or more gestures are detected by the camera to create the selected image/video clips and/or edited image/videos). Paragraph [0058] also teaches that the user interface can incorporate an input module (e.g., a voice command module) configured to receive inputs from the provider for review of content (i.e., detecting one or more gestures), such as receiving inputs from the provider configured to amend and/or highlight aspects of content generated by the scribe (i.e., the data includes medical multimedia data that is manipulated in space responsive to one or more gestures).).
The motivation and rationale for modifying the system for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 1 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 6,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 6 depends on), as described above.
- Shakil further teaches a system, wherein:
- the display includes a transparent lens of smartglasses (Shakil, paragraphs [0020] and [0024]; Paragraph [0020] discloses that the computing device 600 includes a display 112, where the display 112 can be an optical see-through display, an optical see-around display, or a video see-through display (i.e., the display of the augmented reality device may be a transparent lens). Paragraph [0024] teaches that the wearable head-mounted computing device 602 may be VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., the augmented reality device includes examples of smartglasses), and any of the lens elements 610, 612 can be formed of any material (e.g., polycarbonate, CR-39, TRIVEX) that can suitably display a projected image or graphic. Each lens element 610, 612 can also be sufficiently transparent to allow a user to see through the lens element (i.e., the display of the smartglasses includes a transparent lens).).
The motivation and rationale for modifying the system for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 1 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 7,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 7 depends on), as described above.
- Shakil further teaches a system, wherein:
- the portion of the data includes a video, and the feedback data is added to the video when the video is presented at the display (Paragraph [0054] teaches that the provided audio and/or video manipulation tools can facilitate multimedia capture and incorporation of multimedia (e.g., selected image/video clips, edited image/videos) into content generated or prepared by the scribe (e.g., as in multimedia-laden EHR notes) (i.e., the feedback data). In variations of the set of tools including provision of audio and/or video streams to the scribe (i.e., the portion of the data includes a video), the set of tools can also enable to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) to improve performance (i.e., the feedback data is added to the video when the video is presented at the display of the augmented reality device).).
The motivation and rationale for modifying the system for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 1 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 8,
- Shakil et al. (Pub. No. US 2014/0222526) discloses:
- a method to deliver healthcare (Shakil, paragraph [0045]; Paragraph [0045] discloses a method 200 for augmenting performance of a provider.), the method comprising:
- capturing medical multimedia data with one or more sensors, the medical multimedia data being associated with a medical procedure (Shakil, paragraphs [0018], [0019], [0024], and [0025]; Paragraph [0024] discloses that the computer device 600 can be a wearable head-mounted computing device 602, which can be the VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., an augmented reality device which includes one or more sensors). Paragraph [0025] discloses that the video camera 620 can be configured to capture images and may be forward facing to capture at least a portion of the real-world view perceived by the user to generate an augmented reality where computer generated images appear to interact with the real-world view perceived by the user (i.e., capturing the medical multimedia data with the one or more sensors). Paragraphs [0018] and [0019] disclose that these interactions may be interactions between a medical provider and a patient, including conversations between the provider and the patient, wherein the patient provides symptoms, progress, concerns, medication information, allergy information, insurance information, and/or any other suitable health-related information to the provider; transactions wherein the patient provides demographic and/or family history information to the provider; interactions wherein the provider facilitates performance or acquisition of lab tests for the patient; interactions wherein the provider generates image data (e.g., from x-rays, MRIs, CT scanning, ultrasound scanning, etc.) from the patient; interactions wherein the provider generates other health metric data (e.g., cardiology-related data, respiratory data) from the patient; and/or any other suitable interaction between the provider and the patient (see Shakil, paragraph [0018]); an interactive session wherein the provider is examining the patient in a clinical setting or in the examining room of an office or other healthcare facility and eliciting information from the patient by questioning the patient; interactions in a hospital emergency room; interactions in an operating suite where the patient is unconscious; and interactions in a patient’s home, research setting, etc. (see Shakil, paragraph [0019]) (i.e., examples showing that the captured medical multimedia data is medical multimedia in connection with a medical procedure).);
- presenting the medical multimedia data at a display of an augmented reality device (Shakil, paragraph [0024]; Paragraph [0024] discloses that any of the lens elements 610, 612 can be formed of any material (e.g., polycarbonate, CR-39, TRIVEX) that can suitably display a projected image or graphic (i.e., presenting the medical multimedia data at a display of an augmented reality device); and the display capabilities and transparency can facilitate an augmented reality or heads-up display wherein a projected image or graphic is superimposed over a real-world view as perceived by the user through the lens elements 610, 612.);
- transmitting, using a network connection between the augmented reality device and a computing device, the medical multimedia data to the computing device in response to one or more input commands (Shakil, paragraph [0021]; Paragraph [0021] discloses that the computing device 600 preferably enables transmission of data generated using the computing device 600 by way of a communication link 410 (e.g., a wired connection, a wireless connection) that can be configured to communicate with a remote device (i.e., transmitting the medical multimedia data between the augmented reality device and the computing device, using a network connection). Paragraph [0045] discloses that the method 200 includes the provider being able to send a request (i.e., one or more input commands) to transmit at least one of a video stream and an audio stream, from a point of view of the provider during the set of interactions, to a scribe at a scribe cockpit S220; and subsequently transmit the communication from the provider to the scribe cockpit which includes content derived from the set of interactions (i.e., transmitting the medical multimedia data over the network in response to the one or more input commands).);
- obtaining feedback data associated with the medical multimedia data (Shakil, paragraphs [0031] and [0054]; Paragraph [0031] discloses that the scribe cockpit 120 (i.e., an example of a remote device described in paragraph [0021] of Shakil, where the remote device is interpreted to be the equivalent of Applicant’s computing device) enables a scribe to receive information from interactions between a patient and the provider, which can be used to provide guidance and/or feedback to the provider (i.e., obtaining feedback data associated with the medical multimedia). Further, paragraph [0054] discloses that the set of tools [which the scribe is provided] include providing (1) options (e.g., by drop-down menus, by auto-completing partially inputted information) and (2) audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) to facilitate information retrieval for template completion and multimedia capture and incorporation of multimedia into content generated or prepared by the scribe and enables the scribe to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) (i.e., obtaining feedback data associated with the medical multimedia) to improve performance.), …; and
- presenting, at the display of the augmented reality device, the feedback data incorporated with the medical multimedia data (Shakil, paragraph [0029], [0054], [0060], and [0061]; Paragraph [0029] discloses that the computing device 600 (i.e., augmented reality device) with the mobile provider interface 110 allows a provider to summon information from one or more sources, and to receive a response (e.g., at the computing device). The sources can be electronic databases, scheduling systems and tools, electronic information sources (e.g., Wikipedia, PUBMED, UPTODATE, EPOCRATES), and electronic health records (i.e., medical multimedia data), can be mediated by a scribe operating at a scribe cockpit 120. In variations, the response can be provided and/or rendered at a display of a computing device 600 accessible by the provider during interactions with the patient (i.e., displaying the medical multimedia data on the display of the provider’s augmented reality device), and/or during review of content generated by the scribe (i.e., displaying feedback data that is transmitted from the third party device). Further, paragraph [0054] discloses that the scribe tools can also enable a scribe (i.e., a third party) to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) to improve performance. Paragraph [0060] discloses that the feedback to the scribe and/or another entity (e.g., regarding quality of content generated by the scribe) in a qualitative (e.g., free form text/verbal feedback) and/or quantitative (e.g., using a scale of values) manner. Paragraph [0061] discloses that the feedback can be received at an embodiment of the provider workstation and/or the mobile provider interface (i.e., where paragraph [0015] discloses that the mobile provide interface is coupled to the display of the augmented reality device worn by the provider). Since the feedback can be provided to the provider’s computing device in real time in the form of free text or scales of values, this disclosure is interpreted as being the equivalent of providing the feedback data simultaneously with the other health information from various source described in paragraph [0029] (i.e., the medical multimedia data) (also see paragraph [0015] and Figure 1 where the mobile provider interface 110 is coupled to the display 112 worn by the provider). Therefore, Shakil explicitly discloses that the feedback data is displayed simultaneously on the provider’s display on the provider’s goggles, in real-time, during the medical encounter with the health information from the various sources in paragraph [0029].).).
- Shakil does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Curl (Pub. No. WO 2009/137688) teaches a method, wherein:
- the feedback data includes a moving cursor, within an image or a video viewable on the augmented reality device, to indicate a portion of the image or the video (Curl, paragraphs [0042], [0073], and [0074]; Paragraph [0074] teaches that a senior or first physician can be located in a control room or remote location while a second physician, e.g., another physician, fellow or Physician’s Assistant, carries out an invasive procedure in an operating room or procedure room. The first physician can monitor the procedure and communicate with the second physician via audio and graphical mode (i.e., provide feedback), e.g., voice communication over the audio communication subsystem and annotations which are displayed on the main display device 120. The first physician can point to and identify specific items, e.g., features of anatomy, data displayed from various monitoring equipment, vital signs, etc., which are displayed on the main display 120 (i.e., pointing to and identifying specific items on the display control room which are displayed on the main display naturally the feedback includes “moving a cursor to indicate a portion of the image or the video”). The first physician can make the annotations on an auxiliary display 205 located in the control room or remote location, yet these markings will be simultaneously displayed in the operating room (i.e., the feedback data is within an image or a video viewable on the augmented reality device). Paragraph [0042] also explicitly teaches that the integration system 100 provides a cursor on the main high-resolution video display 120 which can be moved and altered using the touchpad 242, keyboard 202, and/or mouse controller 204 located in the control room, and this can allow a control-room participant to draw the attention of an operating-room participant to particular data displayed on the main high- resolution video display 120 (i.e., moving a cursor to indicate a portion of the image or the video). Paragraphs [0073] and [0074] teach that this feature is beneficial for pointing or drawing attention to particular items on a display device and is useful for providing assistance, instructional, oversight, clinical review, or analytical purposes.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to modify the method for augmenting performance of a provider taught by Shakil, to incorporate a step and feature directed to providing feedback data that includes a moving cursor within an image or video viewable on the augmented reality device to indicate a portion of the image or the video, as taught by Curl, in order to point to or draw attention to particular items on a display device and provide assistance, instructions, oversight, clinical review, or analytical purposes. See Curl, paragraph [0073] and [0074]; see also MPEP § 2143 G.
Regarding claim 10,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 8 (which claim 10 depends on), as described above.
- Shakil further teaches a method, wherein:
- the presenting of the medical multimedia data at the display of the augmented reality device includes superimposing the medical multimedia data over a real-world view through a transparent lens of the augmented reality device (Shakil, paragraphs [0020] and [0024]; Paragraph [0024] teaches that combining displaying capabilities and transparency can facilitate an augmented reality or heads-up display wherein a projected image or graphic is superimposed over a real-world view as perceived by the user through the lens elements 610, 612 (i.e., presenting the medical multimedia data at the display of the augmented reality device by superimposing the medical multimedia data over a real-word view of the augmented reality device).), where paragraph [0020] teaches that each lens element 610, 612 can also be sufficiently transparent to allow a user to see through the lens element (i.e., the display of the augmented reality device is a transparent lens).).
The motivation and rationale for modifying the method for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 8 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 11,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 8 (which claim 11 depends on), as described above.
- Shakil further teaches a method, wherein:
- capturing a manipulation of the medical multimedia data in space using the one or more sensors (Shakil, paragraph [0054]; Paragraph [0054] teaches that the system includes a set of tools which provide audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) (i.e., capturing a manipulation of the medical multimedia data in space using the one or more sensors) to facilitate information retrieval for template completion.), the manipulation being captured based on an interpretation of a gesture via the one or more sensors to provide one or more input commands to the augmented reality device (Shakil, paragraph [0054]; Paragraph [0054] teaches that providing audio and/or video manipulation tools can facilitate multimedia capture and incorporation of multimedia (e.g., selected image/video clips, edited image/videos) into content (i.e., the manipulation is captured based on an interpretation of a gesture, being the input command from the user, via the one or more sensors, being the an input module).).
The motivation and rationale for modifying the method for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 8 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 12,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 8 (which claim 12 depends on), as described above.
- Shakil further teaches a method, wherein:
- the medical multimedia data includes a video (Shakil, paragraph [0045]; Paragraph [0045] teaches that the method includes transmitting a request and at least one of a video stream and audio stream from a point of view of the provider during the set of interactions [with the patient] (i.e., the medical multimedia data includes a video), to a scribe at a scribe cockpit S220.), and the feedback data created by the computing device is incorporated into and displayed simultaneously with the video (Shakil, paragraph [0029], [0054], [0060], and [0061]; Paragraph [0029] teaches that the computing device 600 (i.e., augmented reality device) with the mobile provider interface 110 allows a provider to summon information from one or more sources, and to receive a response (e.g., at the computing device). The sources can be electronic databases, scheduling systems and tools, electronic information sources (e.g., Wikipedia, PUBMED, UPTODATE, EPOCRATES), and electronic health records (i.e., medical multimedia data), can be mediated by a scribe operating at a scribe cockpit 120. In variations, the response can be provided and/or rendered at a display of a computing device 600 accessible by the provider during interactions with the patient (i.e., displaying the medical multimedia data on the display of the provider’s augmented reality device), and/or during review of content generated by the scribe (i.e., displaying feedback data that is transmitted from the third party device). Further, paragraph [0054] teaches that the scribe tools can also enable a scribe (i.e., a third party) to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) to improve performance. Paragraph [0060] teaches that the feedback to the scribe and/or another entity (e.g., regarding quality of content generated by the scribe) in a qualitative (e.g., free form text/verbal feedback) and/or quantitative (e.g., using a scale of values) manner. Paragraph [0061] teaches that the feedback can be received at an embodiment of the provider workstation and/or the mobile provider interface. Since the feedback can be provided to the provider’s computing device in real time in the form of free text or scales of values, this disclosure is interpreted as being the equivalent of providing the feedback data simultaneously with the other health information from various source described in paragraph [0029] (i.e., the medical multimedia data) (also see paragraph [0015] and Figure 1 where the mobile provider interface 110 is coupled to the display 112 worn by the provider). Therefore, Shakil explicitly discloses that the feedback data is displayed simultaneously on the provider’s display on the provider’s goggles, in real-time, during the medical encounter with the health information from the various sources in paragraph [0029].).
The motivation and rationale for modifying the method for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 8 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 13,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 8 (which claim 12 depends on), as described above.
- Shakil further teaches a method, wherein:
- the one or more sensors includes a microphone (Shakil, paragraph [0020]; Paragraph [0020] teaches that the computing device includes an audio sensor (i.e., a microphone).) and a camera (Shakil, paragraph [0025]; Paragraph [0025] teaches that the computing device 600 includes a video camera 620.), and the camera is configured to capture the medical multimedia data in response to a verbal command received at the microphone (Shakil, paragraphs [0048] and [0058]; Paragraph [0058] generally teaches that the user interface can incorporate a display configured to present information to the provider, and an input module (e.g., keyboard, mouse, touchpad, touchscreen, voice command module, etc.) configured to receive inputs from the provider for review of content (i.e., receiving verbal commands through the voice command module), and is capable of receiving inputs from the provider configured to amend and/or highlight aspects of content generated by the scribe (i.e., receiving verbal commands to capture the medical multimedia data). For example, paragraph [0048] teaches that the provider can interface with the computing device verbally. In the specific examples of Block S210, the provider can request to pull information from an EHR (e.g., the provider can request cell counts and other metrics related to the patient's health from an EHR), wherein the request is performed by a combination of verbal commands (i.e., capturing the medical multimedia data in response to a verbal command received at the microphone).).
The motivation and rationale for modifying the method for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 8 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 14,
- Shakil et al. (Pub. No. US 2014/0222526) discloses:
- a non-transitory computer-readable storage media storing computer-executable instructions that, when executed by one or more processors, cause operations comprising (Shakil, paragraph [0064]; Paragraph [0064] discloses that various processes of the preferred method can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions (i.e., a non-transitory computer-readable storage medium), where the instructions are preferably executed by computer-executable components preferably integrated with a system and one or more portions of the control module 155 and/or a processor (i.e., the instructions are executed by one or more processors).):
- capturing medical multimedia data with one or more sensors, the medical multimedia data being associated with a medical procedure (Shakil, paragraphs [0018], [0019], [0024], and [0025]; Paragraph [0024] discloses that the computer device 600 can be a wearable head-mounted computing device 602, which can be the VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., an augmented reality device which includes one or more sensors). Paragraph [0025] discloses that the video camera 620 can be configured to capture images and may be forward facing to capture at least a portion of the real-world view perceived by the user to generate an augmented reality where computer generated images appear to interact with the real-world view perceived by the user (i.e., capturing the medical multimedia data with the one or more sensors). Paragraphs [0018] and [0019] disclose that these interactions may be interactions between a medical provider and a patient, including conversations between the provider and the patient, wherein the patient provides symptoms, progress, concerns, medication information, allergy information, insurance information, and/or any other suitable health-related information to the provider; transactions wherein the patient provides demographic and/or family history information to the provider; interactions wherein the provider facilitates performance or acquisition of lab tests for the patient; interactions wherein the provider generates image data (e.g., from x-rays, MRIs, CT scanning, ultrasound scanning, etc.) from the patient; interactions wherein the provider generates other health metric data (e.g., cardiology-related data, respiratory data) from the patient; and/or any other suitable interaction between the provider and the patient (see Shakil, paragraph [0018]); an interactive session wherein the provider is examining the patient in a clinical setting or in the examining room of an office or other healthcare facility and eliciting information from the patient by questioning the patient; interactions in a hospital emergency room; interactions in an operating suite where the patient is unconscious; and interactions in a patient’s home, research setting, etc. (see Shakil, paragraph [0019]) (i.e., examples showing that the captured medical multimedia data is medical multimedia in connection with a medical procedure).);
- transmitting the medical multimedia data to a remote computing device using one or more network connections (Shakil, paragraph [0021]; Paragraph [0021] discloses that the computing device 600 preferably enables transmission of data generated using the computing device 600 by way of a communication link 410 (e.g., a wired connection, a wireless connection) that can be configured to communicate with a remote device (i.e., transmitting the medical multimedia data between the augmented reality device and a remote computing device, using a network connection). Paragraph [0045] discloses that the method 200 includes the provider being able to send a request (i.e., one or more input commands) to transmit at least one of a video stream and an audio stream, from a point of view of the provider during the set of interactions, to a scribe at a scribe cockpit S220; and subsequently transmit the communication from the provider to the scribe cockpit which includes content derived from the set of interactions (i.e., transmitting the medical multimedia data over the network in response to the one or more input commands).);
- obtaining feedback data associated with the medical multimedia data (Shakil, paragraphs [0031] and [0054]; Paragraph [0031] discloses that the scribe cockpit 120 (i.e., an example of a remote device described in paragraph [0021] of Shakil, where the remote device is interpreted to be the equivalent of Applicant’s computing device) enables a scribe to receive information from interactions between a patient and the provider, which can be used to provide guidance and/or feedback to the provider (i.e., obtaining feedback data associated with the medical multimedia). Further, paragraph [0054] discloses that the set of tools [which the scribe is provided] include providing (1) options (e.g., by drop-down menus, by auto-completing partially inputted information) and (2) audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) to facilitate information retrieval for template completion and multimedia capture and incorporation of multimedia into content generated or prepared by the scribe and enables the scribe to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) (i.e., obtaining feedback data associated with the medical multimedia) to improve performance.), …; and
- outputting the feedback data incorporated with the medical multimedia data for display using a display device of the augmented reality device (Shakil, paragraph [0029], [0054], [0060], and [0061]; Paragraph [0029] discloses that the computing device 600 (i.e., the augmented reality device) with the mobile provider interface 110 allows a provider to summon information from one or more sources, and to receive a response (e.g., at the computing device). The sources can be electronic databases, scheduling systems and tools, electronic information sources (e.g., Wikipedia, PUBMED, UPTODATE, EPOCRATES), and electronic health records (i.e., medical multimedia data), can be mediated by a scribe operating at a scribe cockpit 120. In variations, the response can be provided and/or rendered at a display of a computing device 600 accessible by the provider during interactions with the patient (i.e., displaying the medical multimedia data on the display of the provider’s augmented reality device), and/or during review of content generated by the scribe (i.e., displaying feedback data that is transmitted from the third party device). Further, paragraph [0054] discloses that the scribe tools can also enable a scribe (i.e., a third party) to provide real time and/or delayed feedback to the provider regarding aspects of the interactions with the patient (e.g., bedside manner comments) to improve performance. Paragraph [0060] discloses that the feedback to the scribe and/or another entity (e.g., regarding quality of content generated by the scribe) in a qualitative (e.g., free form text/verbal feedback) and/or quantitative (e.g., using a scale of values) manner. Paragraph [0061] discloses that the feedback can be received at an embodiment of the provider workstation and/or the mobile provider interface (i.e., where paragraph [0015] discloses that the mobile provide interface is coupled to the display of the augmented reality device worn by the provider). Since the feedback can be provided to the provider’s computing device in real time in the form of free text or scales of values, this disclosure is interpreted as being the equivalent of outputting the feedback data simultaneously and incorporated with the other health information from various source described in paragraph [0029] (i.e., the medical multimedia data) (also see paragraph [0015] and Figure 1 where the mobile provider interface 110 is coupled to the display 112 worn by the provider). Therefore, Shakil explicitly discloses that the feedback data is displayed simultaneously on the provider’s display on the provider’s goggles, in real-time, during the medical encounter with the health information from the various sources in paragraph [0029].).).
- Shakil does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Curl (Pub. No. WO 2009/137688) teaches a non-transitory computer-readable storage medium, wherein:
- the feedback data includes a moving cursor, within an image or a video viewable on the augmented reality device, to indicate a portion of the image or the video (Curl, paragraphs [0042], [0073], and [0074]; Paragraph [0074] teaches that a senior or first physician can be located in a control room or remote location while a second physician, e.g., another physician, fellow or Physician’s Assistant, carries out an invasive procedure in an operating room or procedure room. The first physician can monitor the procedure and communicate with the second physician via audio and graphical mode (i.e., provide feedback), e.g., voice communication over the audio communication subsystem and annotations which are displayed on the main display device 120. The first physician can point to and identify specific items, e.g., features of anatomy, data displayed from various monitoring equipment, vital signs, etc., which are displayed on the main display 120 (i.e., pointing to and identifying specific items on the display control room which are displayed on the main display naturally the feedback includes “moving a cursor to indicate a portion of the image or the video”). The first physician can make the annotations on an auxiliary display 205 located in the control room or remote location, yet these markings will be simultaneously displayed in the operating room (i.e., the feedback data is within an image or a video viewable on the augmented reality device). Paragraph [0042] also explicitly teaches that the integration system 100 provides a cursor on the main high-resolution video display 120 which can be moved and altered using the touchpad 242, keyboard 202, and/or mouse controller 204 located in the control room, and this can allow a control-room participant to draw the attention of an operating-room participant to particular data displayed on the main high- resolution video display 120 (i.e., moving a cursor to indicate a portion of the image or the video). Paragraphs [0073] and [0074] teach that this feature is beneficial for pointing or drawing attention to particular items on a display device and is useful for providing assistance, instructional, oversight, clinical review, or analytical purposes.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to modify the non-transitory computer-readable medium for augmenting performance of a provider taught by Shakil, to incorporate a step and feature directed to providing feedback data that includes a moving cursor within an image or video viewable on the augmented reality device to indicate a portion of the image or the video, as taught by Curl, in order to point to or draw attention to particular items on a display device and provide assistance, instructions, oversight, clinical review, or analytical purposes. See Curl, paragraph [0073] and [0074]; see also MPEP § 2143 G.
Regarding claim 15,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 14 (which claim 15 depends on), as described above.
- Shakil further teaches a non-transitory computer-readable storage medium, wherein:
- the augmented reality device includes smartglasses, and the one or more sensors form at least part of a sensor assembly of the smartglasses (Shakil, paragraphs [0020] and [0024]; Paragraph [0020] teaches that the computing device 600 includes a display 112, where the display 112 can be an optical see-through display, an optical see-around display, or a video see-through display. Paragraph [0024] teaches that the wearable head-mounted computing device 602 may be VUZIX M100 video eyewear device, Google Glass, Looxcie wearable camera device, a virtual reality headset (e.g., Oculus Rift), and/or any other similar head-mounted display device or wearable augmented reality device (i.e., the augmented reality device includes examples of smartglasses), and any of the lens elements 610, 612 can be formed of any material (e.g., polycarbonate, CR-39, TRIVEX) that can suitably display a projected image or graphic. Each lens element 610, 612 can also be sufficiently transparent to allow a user to see through the lens element (i.e., the one or more sensors form at least part of a sensor assembly of the smartglasses).).
The motivation and rationale for modifying the non-transitory computer-readable medium for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 14 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 17,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 14 (which claim 17 depends on), as described above.
- Shakil further teaches a non-transitory computer-readable storage medium, wherein:
- the capturing the medical multimedia data is responsive to detecting, with the one or more sensors, a first gesture (Shakil, paragraph [0054]; Paragraph [0054] teaches that the system includes a set of tools which provide audio and/or video manipulation tools (e.g., rewind, fast forward, pause, accelerated playback, decelerated playback tools) that are controlled by an input module (e.g., mouse, keyboard, touchpad, foot pedals, etc.) (i.e., a first gesture is detected with the one or more sensors) to facilitate information retrieval for template completion. Paragraph [0054] discloses that providing audio and/or video manipulation tools can facilitate multimedia capture and incorporation of multimedia (e.g., selected image/video clips, edited image/videos) into content (i.e., capturing the medical multimedia data in response to detecting the first gesture).).; and
- the transmitting of the medical multimedia data is responsive to detecting, with the one or more sensors, a second gesture (Shakil, paragraph [0048]; Paragraph [0048] teaches that the provider can interface with the computing device verbally. In the specific examples of Block S210, the provider can request to pull information from an EHR (e.g., the provider can request cell counts and other metrics related to the patient's health from an EHR), wherein the request is performed by a combination of verbal commands (i.e., transmitting the medical multimedia data in response to detecting a second gesture with the one or more sensors).).
The motivation and rationale for modifying the non-transitory computer-readable medium for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 14 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 18,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 14 (which claim 18 depends on), as described above.
- Shakil further teaches a non-transitory computer-readable storage medium, wherein:
- the feedback data includes results of a measurement performed based on the medical multimedia data (Shakil, paragraph [0063]; Paragraph [0063] teaches that the method 200 can facilitate the provider in measuring features of a patient encountered during diagnosis or treatment (e.g., incision dimensions, tissue morphological dimensions, etc.) (i.e., the feedback data that is received from other computing devices will be based on medical multimedia data that includes results of a measurement performed based on the medical multimedia data).).
The motivation and rationale for modifying the non-transitory computer-readable medium for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 14 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Regarding claim 19,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 18 (which claim 19 depends on), as described above.
- Shakil further teaches a non-transitory computer-readable storage medium, wherein:
- the measurement includes a distance measurement on a patient (Shakil, paragraph [0063]; Paragraph [0063] teaches that the method 200 can facilitate the provider in measuring features of a patient encountered during diagnosis or treatment (e.g., incision dimensions, tissue morphological dimensions, etc.) (i.e., measuring the dimensions features of the patient, such as incision dimensions or tissue morphological dimensions, naturally includes a distance measurement on a patient, i.e., measuring the length, width, and/or height of an incision).).
The motivation and rationale for modifying the non-transitory computer-readable medium for augmenting performance of a provider taught by Shakil, in view of Curl, described in the obviousness rejection of claim 14 above similarly apply to this obviousness rejection, and are incorporated herein by reference.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over:
- The combination of: Shakil et al. (Pub. No. US 2014/0222526), as modified in view of Curl (Pub. No. WO 2009/137688), as applied to claims 1 and 8 above, and further in view of:
- Zhang et al. (Pub. No. US 2017/0064214).
Regarding claim 3,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 1 (which claim 3 depends on), as described above.
- Shakil further teaches a system, wherein:
- the augmented reality device includes one or more sensors configured to detect one or more gestures (Shakil, paragraph [0023]; Paragraph [0023] teaches that the computing device 600 can include sensors and elements for eye-tracking and gestural detection (e.g., wink detection) (i.e., one or more sensors configured to detect one or more gestures).).
- The combination of: Shakil, as modified in view of Curl, does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Zhang et al. (Pub. No. US 2017/0064214) teaches a system, wherein:
- the camera is turned on in response to detecting the one or more gestures via the one or more sensors (Zhang, paragraphs [0108] and [0388]; Paragraph [0388] teaches that the image capturing apparatus 100 starts a camera application up. The camera application may be started up according to a user operation. For example, if it is detected that the user clicks an icon of the camera application, the camera application is started up (i.e., the camera is turned on in response to detecting one or more gestures via the one or more sensors). Alternatively, if a voice command used to start the camera application up is detected, the camera application may be started up (i.e., the camera is turned on in response to detecting one or more gestures via the one or more sensors). Paragraph [0108] teaches that this feature is beneficial for performing image signal processing for image quality improvement.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to further modify the method for augmenting performance of a provider taught by Shakil, as modified in view of Curl, to incorporate a step and feature directed to turning on the camera in response to detecting one or more gestures via the one or more sensors, as taught by Zhang, in order to perform image signal processing for image quality improvement. See Zhang, paragraph [0108]; see also MPEP § 2143 G.
Regarding claim 9,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 8 (which claim 9 depends on), as described above.
- The combination of: Shakil, as modified in view of Curl, does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Zhang et al. (Pub. No. US 2017/0064214) teaches a method, further comprising:
- detecting, with one or more sensors, a gesture to turn on the display of the augmented reality device (Zhang, paragraphs [0108] and [0388]; Paragraph [0388] teaches that the image capturing apparatus 100 starts a camera application up. The camera application may be started up according to a user operation. For example, if it is detected that the user clicks an icon of the camera application, the camera application is started up (i.e., turning on the camera is interpreted as turning on the display of the augmented reality device in response to detecting one or more gestures via the one or more sensors). Alternatively, if a voice command used to start the camera application up is detected, the camera application may be started up (i.e., turning on the camera is interpreted as turning on the display of the augmented reality device in response to detecting one or more gestures via the one or more sensors). Paragraph [0108] teaches that this feature is beneficial for performing image signal processing for image quality improvement.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to further modify the system for augmenting performance of a provider taught by Shakil, as modified in view of Curl, to incorporate a step and feature directed to turning on the display of the augmented reality device in response to detecting one or more gestures via the one or more sensors, as taught by Zhang, in order to perform image signal processing for image quality improvement. See Zhang, paragraph [0108]; see also MPEP § 2143 G.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over:
- The combination of: Shakil et al. (Pub. No. US 2014/0222526), as modified in view of Curl (Pub. No. WO 2009/137688), as applied to claim 14 above, and further in view of:
- Doo et al. (Pub. No. US 2017/0042631).
Regarding claim 16,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 14 (which claim 16 depends on), as described above.
- The combination of: Shakil, as modified in view of Curl, does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Doo et al. (Pub. No. US 2017/0042631) teaches a non-transitory computer-readable storage medium, wherein:
- the operations include converting at least the portion of the medical multimedia data into a compatible form associated with an operating system of the remote computing device (Doo, paragraph [0066]; Paragraph [0066] teaches that the input codec 74 can receive the image file 42 from the image source 44 and decompress the image file 42. If the image defined by the image file 42 is not to be modified or analyzed, the decompressed image file 42 can be transmitted to the transcoder 76. The transcoder 76 can convert to the image file 42 to a different format of similar or like quality to gain compatibility with another program or application, if necessary (i.e., converting at least the portion of the medical multimedia data into a compatible form associated with an operating system of the remote computing device). Paragraph [0066] teaches that this feature is beneficial for gaining compatibility with another program or application.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to further modify the non-transitory computer-readable storage medium for augmenting performance of a provider taught by Shakil, as modified in view of Curl, to incorporate a step and feature directed to converting at least the portion of the medical multimedia data into compatible form associated with an operating system of the remote computing device, as taught by Doo, in order to gain compatibility with another program or application. See Doo, paragraph [0066]; see also MPEP § 2143 G.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over:
- The combination of: Shakil et al. (Pub. No. US 2014/0222526), as modified in view of Curl (Pub. No. WO 2009/137688), as applied to claim 14 above, and further in view of:
- Saget et al. (Pub. No. US 2019/0122330).
Regarding claim 20,
- The combination of: Shakil, as modified in view of Curl, teaches the limitations of claim 14 (which claim 20 depends on), as described above.
- Shakil teaches a non-transitory computer-readable storage medium, wherein:
- detecting one or more gestures with the one or more sensors (Shakil, paragraph [0023]; Paragraph [0023] teaches that the computing device 600 can include sensors and elements for eye-tracking and gestural detection (e.g., wink detection) (i.e., detecting one or more gestures with the one or more sensors).).
- The combination of: Shakil, as modified in view of Curl, does not explicitly teach, however, in analogous art of systems and methods which utilize augmented reality devices, Saget et al. (Pub. No. US 2019/0122330) teaches a non-transitory computer-readable storage medium, wherein:
- storing the medical multimedia data at a storage location responsive to
detecting the one or more gestures (Saget, paragraphs [0086] and [0092]; Paragraph [0086] teaches that the system 1 allows the user 155 to see critical work information right in their field-of-image using a see-through visual display and then interact with it using familiar gestures, voice commands, and motion tracking (i.e., the one or more gestures). The data can be stored in data storage (i.e., storing the medical multimedia data at a storage location responsive to detecting the one or more gestures). Paragraph [0092] teaches that this feature is beneficial for storage and retrieval of preoperative medical images, including any metadata associated with these images and the ability to query those metadata.).
Therefore, it would have been obvious to one of ordinary skill in the art of systems and methods which utilize augmented reality devices at the time of the effective filing date of the claimed invention to further modify the non-transitory computer-readable storage medium for augmenting performance of a provider taught by Shakil, as modified in view of Curl, to incorporate a step and feature directed to storing the medical multimedia data at a storage location responsive to detecting the one or more gestures, as taught by Saget, in order to store and retrieve preoperative medical images, including any metadata associated with these images and the ability to query those metadata. See Saget, paragraph [0092]; see also MPEP § 2143 G.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nicholas Akogyeram II whose telephone number is (571) 272-0464. The examiner can normally be reached Monday - Friday, between 8:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Dunham can be reached at (571) 272-8109. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Official replies to this Office action may now be submitted electronically by registered
users of the EFS-Web system. Information on EFS-Web tools is available on the Internet at:
http://www.uspto.gov/patents/processlfi!elefslguidance/index.isp. An EFS-Web Quick-Start
Guide is available at: http://www.uspto.gov/ebc/portallefslquick-start.pdf.
Alternatively, official replies to this Office Action may still be submitted by any one of fax, mail, or hand delivery.
Faxed replies should be directed to the central fax at (571) 273-8300.
Mailed replies should be addressed to:
United States Patent and Trademark Office:
Commissioner of Patents and Trademarks
P.O. Box 1450
Alexandria, VA 22313-1450
Hand delivered responses should be brought to the United States Patent and Trademark Office Customer Service Window:
Randolph Building
401 Dulany Street
Alexandria, VA 22314-1450
/N.A.A./Examiner, Art Unit 3686
/JONATHON A. SZUMNY/Primary Examiner, Art Unit 3686
Prosecution Insights