Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to amendments
In the amendments dated 6/11/26, Applicant has amended claims 1 and 6, and cancelled claim 13. Claims 1-12, 14-18 are pending.
The amendments overcome the previous 112(b) rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-12, 14-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 has been amended to recite ““instead of reflecting a real-time image of a hand of the user.” However, support for this limitation has not been found in the specification.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-12, 14-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang [US 2017/0258526 A1] in view of Joskowicz et al. [US 2009/0177081 A1, hereinafter “Joskowicz”], and further in view of Kesavadas et al. [US 2013/0288214 A1, hereinafter “Kesavadas,” newly cited].
Re. claim 1, Lang discloses:
An information processing system comprising a controller including at least one processor [Par. 0072] configured to execute each of following steps including:
a reading step of reading reference information on a reference maneuver [“virtual surgical plan,” Par. 0147] that is a medical maneuver to be referred to by a user, the reference information including a three-dimensional position of a track point in the reference maneuver [“The multiple OHMD's can be registered in a common coordinate system 15 using anatomic structures, anatomic landmarks, calibration phantoms, reference phantoms, optical markers, navigation markers, and/or spatial anchors, for example like the spatial anchors used by the Microsoft Hololens. Pre-operative data 16 of the patient can also be registered in the common coordinate system 15….The live data 18 of the patient can be registered in the common coordinate system 15. Intra-operative imaging studies 20 can be registered in the common coordinate system 15. OR references, e.g. an OR table or room fixtures can be registered in the common coordinate system 15 using, for example, optical markers IMU's, navigation markers or spatial mapping 22,” Par. 0147. Thus, any of the above objects can be a track point]
and pre-recorded time-series images [“the surgeon wearing OHMD…can execute commands…to display the next predetermined bone cut, e.g. from a virtual surgical plan…which can…project digital holograms of the next surgical step 34 superposed onto and aligned with the surgical site in a predetermined position and/or orientation,” Par. 0147. Because the system has digital holograms of the next surgical step, the virtual surgical plan necessarily includes time series images of the planned procedure which are read by the above processor in order to project said images], each of the time series images including an augmented reality medical instrument used in the reference maneuver [the OHMD displays the” virtual surgical instrument,” Pars. 0073-0074];
a first reception step of receiving a captured image acquired by capturing an image of an external world as viewed by the user [“Live data 18 of the patient, for example from the surgical site…can be measured…using…image or video capture systems,” Par. 0147]; and
a generation step of generating, based on the three-dimensional position of the track point, the pre-recorded time series images, and the captured image, display information for presenting augmented reality regarding the reference maneuver to the user [“The pre-operative data 16 or live data 18 including intra-operative measurements or combinations thereof can be used to … modify a virtual surgical plan 24…[and the system can] display the next predetermined bone cut, e.g. from a virtual surgical plan or an imaging study or intra-operative measurements, which can trigger the OHMD's 11, 12, 13, 14 to project digital holograms of the next surgical step 34 superimposed onto and aligned with the surgical site in a predetermined position and/or orientation,” Par. 0147] the augmented reality three-dimensionally reproducing an aspect of the augmented reality medical instrument [“the OHMD can display…[a] virtual surgical instrument,” Par. 0074 at a variety of orientations for a plurality of users, i.e. in three dimensions; Par. 0147] used in the reference maneuver
Regarding the reference maneuver being performed by a person different from a user and including pre-recorded time series images captured by an imaging device during an actual physical performance of the reference maneuver by the person, Lang is silent. However, Kesavadas teaches, in an information processing system including a controller, a processor configured to read a reference maneuver performed by a person different from a user [the teaching video, “a video of a …surgery…performed by a surgeon,” Par. 0034] the reference information including pre-recorded time series images captured by an imaging device during an actual physical performance of the reference maneuver by the person [“live video of a surgery being performed in real time…compris[ing] a plurality of frames,” Par. 0034], including a captured appearance of a surgical tool [“one of the …tools is visible,” Par. 0034]. Kesavadas further teaches generating display information for displaying AR regarding the reference maneuver, the AR reproducing the captured appearance of the instrument by retrieving and displaying the captured appearance of the instrument used by the person from the pre-recorded time series images [Fig. 10]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the apparatus of Lang by recording, for the reference maneuver, steps performed in an actual physical performance of the reference maneuver by a different person, as taught by Kesavadas, because this allows the system to be used in guiding and training a surgeon in both cognitive and motor skill areas using a training video performed by a more senior surgeon [Pars. 0008-0011].
Lang teaches that “another medical instrument” is the actual physical medical instrument [Par. 0073] but fails to disclose generating display information for presenting augmented reality reproducing a hand manipulating the (physical) medical instrument. However, Joskowicz teaches, in a surgical system having a processor and display, the system’s processor is configured to generate, based on a position of a surgeon’s hand, display information for presenting augmented reality reproducing a hand manipulating a medical instrument [Par. 0039: “the video camera system could be arranged to provide an augmented virtuality display of the patient's … and a real-time image of the surgeon's hand holding the robot base or a targeting jig is imposed on this virtual image of the patient's head”]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system of Lang to configure the processor to generate an image of a hand manipulating the physical medical instrument, as taught by Joskowicz, in order to “guid[e] the surgeon in positioning” the instrument [Joskowicz, Par. 0039].
Regarding the limitations of the reference maneuver including “a captured appearance of a hand of the person manipulating the medical instrument,” and the generation step reproducing an aspect of the AR medical instrument used int eh reference maneuver and the “captured appearance of the hand” manipulating the medical device, and “retrieving and displaying the captured appearance of the hand…,” given the above teachings, one of ordinary skill would reasonably be apprised of the benefits of including the hand (because the surgery of Lang and Joskowicz may include a hand holding the instrument) in the teaching video.
Regarding the limitation “instead of reflecting a real-time image of a hand of the user,” depending on the type of display, it may not be necessary to reflect the real-time image of the hand, which may be directly visible (in the case of, for example, AR glasses). Thus, displaying the recorded image of the hand rather than the real time image would have been within the scope of the abilities of one of ordinary skill, and they would have reasonably been apprised of the benefits of such.
Re. Claim 2, Lang discloses the track point is set on the another (physical) medical instrument [“optical markers…can…be attached to the instrument,” Par. 1548], and the three-dimensional position is a relative position with respect to a surgical field of the reference maneuver [Par. 0384]. Lang fails to teach the track point being set on the hand manipulating the another medical instrument. However, given the above teachings, in which the hand is displayed, It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the system such that the track point is placed on the hand manipulating said medical instrument, because in order for a real-time image of the hand to be displayed, it must be tracked in some manner; so setting this to be the track point is an obvious modification readily performed by one of ordinary skill in the art, given the above-modified system.
Re. Claim 3, Lang discloses the processor is configured to further execute:
a first specification step of identifying a location of a surgical field visible to the user in the captured image [see e.g. Par. 0259, which describes “field of view” tracking, and registration, Par. 0260]; and
the generation step where the augmented reality medical instrument is superimposed on the surgical field [Par. 1548].
Regarding the hand being superimposed on the surgical field, Joskowicz teaches this as set forth above.
Re. Claim 4, Lang discloses the processor is configured to further execute:
a second specification step of identifying a user maneuver from the captured image, the user maneuver being a medical maneuver performed by the user [Par. 1548 describes how the surgeon’s movements are tracked and registered live, with the virtual display included augmented reality aspect which track this maneuver.]; and
in the generation step, the display information is modified according to an aspect of the user maneuver [Par. 1548 discloses how the physical and virtual tools are aligned in the OHMD, so that e.g. the hidden portions of a rod can be virtually displayed].
Re. Claim 5, Lang discloses The information processing system according to claim 4, wherein, in the augmented reality, progression of the reference maneuver is controlled according to the aspect of the user maneuver [Par. 1549: “pedicle screws and related instruments…or trocars…can be placed reliably following a trajectory or desired position of the…instruments…”].
Re. Claim 6, the modified Lang discloses The information processing system according to claim 5, wherein the augmented reality reproduces the reference maneuver wherein the reproduction represents a time-series future step showing the augmented reality medical instrument as at least one of the pre-recorded time-series images (the AR instrument is displayed as time series images; see claim 1 above) compared with a current aspect of the user maneuver [Par. 1549: “pedicle screws and related instruments…or trocars…can be placed reliably following a …desired position of the…instruments…”].
Re. Claim 7, Lang discloses the augmented reality is configured to be such that a progression speed of the reference maneuver is variable [Par. 1420. Because the surgeon is able to toggle between steps, the surgeon can control/vary the progression speed of the reference maneuver.].
Re. Claim 8, Lang discloses the augmented reality is configured to be such that transparency is variable of the augmented reality medical instrument and the hand manipulating the another medical instrument [“The surgeon can optionally adjust the transparency or opacity of the virtual data displayed in the OHMD,” Par. 0210]. Regarding the hand being superimposed on the surgical field, Joskowicz teaches this as set forth above.
Re. Claim 9, Lang discloses the augmented reality presents, to the user, time-series changes in the aspect of the augmented reality medical instrument, with time-series changed aspects superimposed on each other [Par. 1421]. Regarding the hand being superimposed on the surgical field, Joskowicz teaches this as set forth above, and the above teachings would reasonably convey, to one of ordinary skill, the benefits of time-series changes in the aspect of the hand being superimposed as well.
Re. Claim 10, Lang discloses each of the time-series changed aspects is based on a discretely selected frame [Par. 1420].
Re. Claim 11, Lang discloses the reading step reads the reference information, which is stored in a memory in advance, is read from the memory [Par. 0450. The coordinates are read from a memory and a “virtual surgical plan” may be adjusted, updated, or modified. Thus, the reading happens in advance of the actual procedure.].
Re. Claim 12, Lang discloses the processor is configured to further execute: a second reception step of receiving the reference information from outside via a network [Par. 0909]; and in the reading step the reference information received from outside via the network is read [in order to develop the surgical plan using data stored on an outside device via a network, the outside reference information must be read].
Re. Claim 14, Lang discloses the generation step further generates information to be perceived by the user with a sense other than sight [audible alarm, Par. 1360].
Re. Claim 15, Lang discloses a display, the display being configured to: transmit light from the external world in a direction toward the user [Par. 1547]; and display a screen based on the display information so that the user can view a state where the reference maneuver is superimposed on the external world is configured to be viewed by the user [Par. 1548].
Re. Claim 16, Lang discloses the display is included in a wearable device wearable for the user [a head-mounted display, Par. 0002].
Re. Claim 17, the modified Lang teaches the method steps of claim 1.
Re. Claim 18, the modified Lang teaches a non-transitory computer-readable memory medium storing the programming of claim 1 [inherently; see storage referred to in e.g. Par. 1259; furthermore, because the steps are executed electronically there must be some storage medium which stores the steps].
Response to Arguments
Applicant's arguments filed 6/11/26 have been fully considered but they are not persuasive. The examiner agrees that Lang fails to explicitly describe a pre-recorded maneuver performed by a different surgeon. However, Kesavadas, as set forth above, teaches the use of such a teaching video and the advantages of such.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIN MCGRATH whose telephone number is (571)270-0674. The examiner can normally be reached M-Th 9 am to 3 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JACKIE HO can be reached at (571) 272-4696. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERIN MCGRATH/Primary Examiner, Art Unit 3771
Prosecution Insights