DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
1. This action is responsive to communications: Amendment, filed on 03/30/2026.
This action is made FINAL.
2. Claims 1-20 are pending in the case. Claims 1, 11 and 20 are independent claims. Claims 1-3, 9, 11-13, 18 and 20 have been amended.
Response to Arguments
Applicant's arguments filed March 30, 2026 have been fully considered but they are not persuasive.
Applicant argues (claims 1, 11 and 20) Makrinich fails to disclose a protocol that specifies for each step of a plurality of steps of the procedure, an expected location, pose, or positional relationship of one or more objects associated with performance of the step.
In response, Makrinich discloses (Para 129) a system 401 may include a database 411 for storing various types of data related to previously conducted surgeries (i.e., historical surgical data that may include historical image, video or audio data, text data, doctors' notes, data obtained by analyzing historical surgical data, and other data relating to historical surgeries). Makrinich (Para 129) discloses the stored historical surgical data is related to surgical procedures (Para 127) that include medical actions (Para 141, 142) associated with surgical instruments position/motion relative to a surface and deviation therefrom (Para 167). Thus, Makrinich discloses a protocol that specifies for each step of a plurality of steps of the procedure, an expected location, pose, or positional relationship of one or more objects associated with performance of the step.
Applicant argues (claims 1, 11 and 20) Makrinich fails to disclose comparing an observed location, pose, or positional relationship of one or more objects derived from the sensor data to the expected location, pose, or positional relationship specified by the protocol for a current step of the plurality of steps of the procedure.
In response, Makrinich discloses tracking motion and evaluate boundary data of the surgical plane to determine projected deviation of the surgical instrument from the surgical plan, such that projected deviation from a surgical plane indicates incorrect operation/action (167, 168). The deviation as taught by Makrinich corresponds to the comparison of a surgical instrument position to the expected position, which Makrinich (Para 142) discloses simplifies the surgical procedure by eliminating unexpected departure, e.g. piercing an organ, from a surgical plane. Thus, Makrinich discloses comparing an observed location, pose, or positional relationship of one or more objects derived from the sensor data to the expected location, pose, or positional relationship specified by the protocol for a current step of the plurality of steps of the procedure.
Applicant argues claims 2-10 and 12-19 are in condition for allowance for at least reasons established with regard to independent claims 1 and 11.
In response, claims 2-10 and 12-19 are not patentable based at least on the above response to Applicant’s Remarks with regard to independent claims 1 and 11.
To the extent that the response to the applicant's arguments may have mentioned new portions of the prior art references which were not used in the prior office action, this does not constitute a new ground of rejection. It is clear that the prior art reference is of record and has been considered entirely by applicant. See In re Boyer, 363 F.2d 455, 458 n.2, 150 USPQ 441, 444, n.2 (CCPA 1966) and In re Bush, 296 F.2d 491, 496, 131 USPQ 263, 267 (CCPA 1961).
The mere fact that additional portions of the same reference may have been mentioned or relied upon does not constitute new ground of rejection. In re Meinhardt, 392, F.2d 273, 280, 157 USPQ 270, 275 (CCPA 1968).
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 11 and 20 recite:
a protocol that specifies…(ii) for each step of a plurality of steps of the procedure, an expected location, pose, or positional relationship of one or more objects…
based on the sensor data, determining, by the interactive procedural system, whether the operator is performing the procedure correctly by comparing an observed location, pose, or positional relationship of one or more objects derived from the sensor data to the expected location, pose, or positional relationship…
Applicant’s Specification has not been to support “positional relationship”. Correction is required.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 11 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential steps, such omission amounting to a gap between the steps. See MPEP § 2172.01. The omitted steps are: the sensor sensing physical objects in its field of view to detect the object and the object attitude and/or orientation as disclosed by Applicant’s Specification (Para 38).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by
Evgeny Makrinich et al., US 2021/0313052 A1.
Independent claim 1, Makrinich discloses a computer-implemented method, comprising: accessing, by an interactive procedural system, a protocol that specifies a procedure to be performed by an operator wearing an augmented, mixed, or extended reality (AR/MR/XR) device (i.e. access surgical procedures – abstract; Para 286– stored in database files - Para 129, 132; provided via augmented reality – Fig. 1) and (ii) for each step of a plurality of steps of the procedure, an expected location, pose, or positional relationship of one or more objects associated with performance of the step (i.e. access historical surgical data related to medical actions and surgical instrument movements – Para 127, 129, 141, 142; track motion and evaluate boundary data of the surgical plane to determine projected deviation of the surgical instrument from the surgical plane – Para 167);
based on the protocol, generating, by the interactive procedural system, instructions to be outputted by the AR/MR/XR device to assist the operator in performing the procedure (i.e. software instructions provide data about surgical events – Para 136, 204; Fig. 8B “E1-EN”);
providing, for output by the interactive procedural system and to the AR/MR/XR device, the instructions (i.e. output an outcome – Para 145; Fig. 8A “815”);
in response to providing, for output, the instructions, receiving, by the interactive procedural system, sensor data that reflects characteristics of an environment where the operator wearing the AR/MR/XR device is located (i.e. camera captures surgery – Para 60; track movement of the surgical instrument to determine deviation from a path or plane – Para 167);
based on the sensor data, determining, by the interactive procedural system, whether the operator is performing the procedure correctly (i.e. camera captures surgery – Para 60; track movement of the surgical instrument to determine deviation from a path or plane – Para 167; provide a warning signal corresponding to a desirable direction of movement – Para 169) by comparing an observed location, pose, or positional relationship of one or more objects derived from the sensor data to the expected location, pose, or positional relationship specified by the protocol for a current step of the plurality of steps of the procedure (i.e. projected deviation from a surgical plane indicates incorrect operation/action – 167, 168);
based on determining whether the operator is performing the procedure correctly, generating, by the interactive procedural system, feedback to be outputted by the AR/MR/XR device to assist the operator in determining whether to adjust actions being performed by the operator in performing the procedure (i.e. a warning signal includes instructions to the display of an interface area; instructions such as directional haptic feedback, illumination of lights or other signals on the surgical instrument corresponding to a desirable direction of movement, may include audible instructions delivered through a speaker or other sound-producing device, or any other communication medium – Para 169); and
providing, for output by the interactive procedural system and to the AR/MR/XR device, the feedback (i.e. control of the surgical instrument via smart glasses - Fig. 1; Para 74 – providing augmented reality display – Para 178, 182).
Claim 2, Makrinich discloses the method of claim 1, wherein: determining whether the operator is performing the procedure correctly comprises determining that the operator is performing the procedure correctly, and generating the feedback comprises: generating a visual indicator that indicates that the operator is performing the procedure correctly (i.e. provide a warning signal corresponding to a desirable direction of movement – Para 169);
identifying a physical object in the environment where the operator wearing the AR/MR/XR device is located (i.e. the camera control application may identify an anatomical structure – Para 61); and
generating instructions to augment the physical object while the physical object is in the field of view of the operator wearing the AR/MR/XR device (i.e. display video clips and procedures in an augmented reality device – Para 178, 212, 213).
Claim 3, Makrinich discloses the method of claim 1, wherein: determining whether the operator is performing the procedure correctly comprises determining that the operator is performing the procedure incorrectly (i.e. provide alerts for potential deviations – Para 170), and generating the feedback comprises:
generating a first visual indicator that indicates that the operator is performing the procedure incorrectly (i.e. provide alerts for potential deviations – Para 170);
identifying a first physical object (i) in the environment where the operator wearing the AR/MR/XR device is located (i.e. the camera control application may identify an anatomical structure – Para 61) and (ii) where an incorrect performance of the procedure occurred (i.e. track movement of the surgical instrument to determine deviation from a path or plane – Para 167, 173);
generating a second visual indicator (i.e. feedback mechanism provides audible or visual instructions – para 169);
identifying a second physical object (i) in the environment where the operator wearing the AR/MR/XR device is located (i.e. determining a current step in a surgical procedure – Para 171) and (ii) where a correction to the incorrect performance should occur (i.e. detecting instrument deviations – Para 173 – via tracking – Para 174 - and providing recommendation/notification – Para 178); and
generating first instructions to augment the first physical object while the first physical object is in the field of view of the operator wearing the AR/MR/XR device (i.e. display video clips and procedures in an augmented reality device – Para 178, 212, 213).and second instructions to augment the second physical object while the second physical object is in the field of view of the operator wearing the AR/MR/XR device (i.e. provide audible or visual instructions on how to manipulate a surgical instrument – Para 169).
Claim 4, Makrinich discloses the method of claim 1, wherein the sensor data are generated by sensors that (i) comprise a camera, a time of flight sensor, a structured illumination sensor, an infrared sensor, a microphone, and a light detection and ranging scanner (i.e. camera including a time of flight camera and light detector and ranging camera – Para 66; Fig. 1, 2), (ii) that are integrated with the AR/MR/XR device, (iii) that are separate from the AR/MR/XR device and integrated with a fixed object (Fig. 2), and (iv) that are separate from the AR/MR/XR device and integrated with an object manipulated by the operator during performance of the procedure (i.e. surgical tool may include cameras – Fig. 3; Para 73).
Claim 5, Makrinich discloses the method of claim 1, wherein a format of the protocol is a csv, xml, or xmlx format (i.e. data/files may be in an XML database – Para 51, 319).
Claim 6, Makrinich discloses the method of claim 1, wherein generating the instructions to be outputted by the AR/MR/XR device to assist the operator in performing the procedure: receiving additional sensor data that reflects additional characteristics of the environment where the operator wearing the AR/MR/XR device is located (i.e. receiving additional verbal interactions – Para 263);
based on the additional sensor data, selecting a step of the procedure (i.e. system identifies surgical actions and recommends step(s) – Para 267, 268, 271); and
based on the step of the procedure, generating the instructions (i.e. recommend step(s) – Para 267, 268, 271).
Claim 7, Makrinich discloses the method of claim 6, wherein the additional sensor data is audio data (i.e. voice/speech input – Para 93; 102).
Claim 8, Makrinich discloses the method of claim 7, wherein selecting the step of the procedure comprises:
determining a spoken term included in the audio data (i.e. receiving speech commands – Para 102);
comparing the spoken term to a set of terms (i.e. using speech recognition algorithms – Para 102);
determining that the spoken term corresponds to a term of the set of terms (i.e. machine learning algorithm includes speech recognition algorithm to correspond inputs with outputs – Para 54); and
selecting the step of the procedure based on the term of the set of terms (i.e. machine learning algorithm estimates outputs for inputs – Para 54; and selects one of the outputs – Para 55).
Claim 9, Makrinich discloses the method of claim 1, wherein determining whether the operator is performing the procedure correctly comprises:
analyzing the sensor data (i.e. use a motion detection algorithm to analyze video frames for deviation from a surgical plane – Para 166, 167, 168);
based on analyzing the sensor data, determining (i) a change in a location of an object within a field of view of the operator wearing the AR/MR/XR device (i.e. track motion and evaluate boundary data of the surgical plane to determine projected deviation of the surgical instrument from the surgical plane – Para 167), (ii) a change in a pose of the object, (iii) a change in a distance between the object and the AR/MR/XR device, or (iv) a change in a distance between the object and an additional object (i.e. analyze distance between the surgical too and a point of an anatomical structure – Para 150); and
determining whether the operator is performing the procedure correctly based on (i) the change in the location of the object within the field of view of the operator wearing the AR/MR/XR device (i.e. projected deviation from a surgical plane indicates incorrect operation/action – 167, 168) (ii) the change in the pose of the object, (iii) the change in the distance between the object and the AR/MR/XR device (i.e. predetermined distance identity to allow/ascertain human error – Para 168), or (iv) the change in the distance between the object and an additional object.
Claim 10, Makrinich discloses the method of claim 1, comprising: receiving additional sensor data that reflects additional characteristics of the environment where the operator wearing the AR/MR/XR device is located; and based on the additional sensor data (i.e. receiving additional verbal interactions – Para 263),
determining the procedure being performed by the operator (i.e. system identifies surgical actions - Para 267, 268, 271), wherein accessing the protocol is based on determining the procedure being performed by the operator wearing the AR/MR/XR device (i.e. recommends step(s) – Para 271; access surgical procedures – abstract; Para 286– stored in database files - Para 129, 132).
Independent claim 11, the claim is similar in scope to claim 1. Therefore, similar rationale as applied in the rejection of claim 1 applies herein.
Claims 12-19, the corresponding rationale as applied in the rejection of claims 2-10 apply herein.
Independent claim 20, the claim is similar in scope to claim 1. Therefore, similar rationale as applied in the rejection of claim 1 applies herein.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANTE HARRISON whose telephone number is (571)272-7659. The examiner can normally be reached Monday - Friday 8:00 am to 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANTE E HARRISON/Primary Examiner, Art Unit 2615
Prosecution Insights