DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 3-9, 11-17, and 19-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-17 of U.S. Patent No. 12293472 in view of Steinbrücker et al. (US 20190197777 A1),
Table I:
Mapping of contending claims in the patent that contains double patenting issues.
Current Application (19/098,396)
Patent (US 12293472 B2) in view of Steinbrücker et al.
1
1
3
2
4
3
5
4
6
5
7
6
8
7
9
8
11
9
12
10
13
11
14
12
15
13
16
14
17
15
19
16
20
17
Table II:
Current Application (19/098,396)
Patent (US 12293472 B2) in view of Steinbrücker et al.
Claim 1, A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
access a model of a recognized object depicted in an original image of a surgical site at which a surgical procedure is being performed using a computer-assisted medical system;
determine, based on kinematic data tracked by the computer-assisted medical system, a pose of the recognized object at the surgical site;
associate the model with the recognized object;
determine, based on the model, that a portion of a synthetic element will occlude at least a portion of the recognized object; and
generate, based on the determining that the portion of the synthetic element will occlude at least the portion of the recognized object, presentation data for use by a presentation system to present an augmented version of the original image in which the synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object.
Claim 1, A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
access a model of a recognized object depicted in an original image of a scene;
Claim 3: … access kinematic data representative of a pose of the recognized object; and identify the recognized object within imagery depicted by the original image based on the kinematic data
associate the model with the recognized object;
Steinbrücker: ¶112, “Occlusion processing identifies portions of a virtual object that should not be rendered for and/or displayed to a user because there is an object in the physical world blocking that user's view of the location where that virtual object is to be perceived by the user. . . The world reconstruction provides a model from which information about objects in the physical world may be obtained for such calculations.”
and
generate presentation data for use by a presentation system to present an augmented version of the original image in which a synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object,
. . .
Claim 1 is rejected for obviousness type double patenting over claim 1 of the patent 12293472 in view of Steinbrücker for having similar limitations as described in Table 2.
The patent 12293472 fails to disclose the data points are “determine, based on the model, that a portion of a synthetic element will occlude at least a portion of the recognized object;” in Claim 1. Steinbrücker teaches “methods and apparatus for quickly generating environments containing computer-generated objects” (¶5). ¶112 teaches occlusion processing based on models associated with the physical objects; and recites “Occlusion processing identifies portions of a virtual object that should not be rendered for and/or displayed to a user because there is an object in the physical world blocking that user's view of the location where that virtual object is to be perceived by the user. . . The world reconstruction provides a model from which information about objects in the physical world may be obtained for such calculations.” Thus, it would have been obvious to one with ordinary skill, in the art at the time of the invention, to modify the method (taught by patent 12293472) to determine the occlusion data based on the model (taught by Steinbrücker). The suggestions/motivations would have been to “present realistic and readily understandable information " (¶4).
Although the conflicting claims are not identical, they are not patentably distinct from each other because the scope of the inventions is the same. Claim 1 of current application is an obvious variant of claim 1 of the patent 12293472 in view of Steinbrücker.
The same logic applies to Claims 3-9, 11-17, and 19-20. They are rejected for obviousness type double patenting under claims 2-17 of the patent 12293472.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5-8, 9, and 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ryan et al. (US 20180168740 A1), and in view of Steinbrücker et al. (US 20190197777 A1).
Regarding Claim 1, Ryan discloses A system (ABST reciting “a mixed reality surgical navigation system (10) to be worn by a user (106) during a surgical procedure”) comprising:
access a model of a recognized object depicted in an original image of a surgical site at which a surgical procedure is being performed using a computer-assisted medical system; (Figs. 31-33 show a surgical site at which a hip replacement surgery is being performed by an instrument. Further, ¶172 reciting “The system may also be used to recognize surgical instruments and implants encountered during surgery. A database of CAD models of instruments and equipment to scale is held in memory. During a procedure, SLAM or similar machine vision algorithms can capture topography of items in the scene and compare to the database on instruments and equipment.”)
determine, based on kinematic data tracked by the computer-assisted medical system, a pose of the recognized object at the surgical site; (¶172 reciting “If a match is found, system can then take actions appropriate such as tracking the position and orientation of instruments relative to the patient and other instruments being used in surgery or enter a mode relevant to use of that instrument.”)
associate the model with the recognized object; (¶172 reciting “During a procedure, SLAM or similar machine vision algorithms can capture topography of items in the scene and compare to the database on instruments and equipment. If a match is found, system can then take actions appropriate such as tracking the position and orientation of instruments relative to the patient and other instruments being used in surgery or enter a mode relevant to use of that instrument.”)
However, Ryan does not explicitly disclose a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
determine, based on the model, that a portion of a synthetic element will occlude at least a portion of the recognized object; and
generate, based on the determining that the portion of the synthetic element will occlude at least the portion of the recognized object, presentation data for use by a presentation system to present an augmented version of the original image in which the synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object.
Steinbrücker teaches “methods and apparatus for quickly generating environments containing computer-generated objects” (¶5). ¶21 recites “at least one non-transitory computer-readable medium comprising computer-executable instructions configured for execution on the CPU that, when executed by the CPU, perform a method.” ¶112 teaches occlusion processing based on models associated with the physical objects; and recites “Occlusion processing identifies portions of a virtual object that should not be rendered for and/or displayed to a user because there is an object in the physical world blocking that user's view of the location where that virtual object is to be perceived by the user. . . The world reconstruction provides a model from which information about objects in the physical world may be obtained for such calculations.” Further, ¶356 recites “The realism with which AR and MR scenes are presented to a user may be enhanced by providing occlusion data to applications generating such scenes”; and ¶363 reciting “the occlusion data may be provided to one or more applications wanting to have virtual objects occluded by real objects”
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the system (taught by Ryan) to do the occlusion processing (taught by Steinbrücker). The suggestions/motivations would have been to The suggestions/motivations would have been to “present realistic and readily understandable information " (¶4), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Claim 9, has similar limitations as of Claim(s) 1, therefore it is rejected under the same rationale as Claim(s) 1.
Claim 17, has similar limitations as of Claim(s) 1, therefore it is rejected under the same rationale as Claim(s) 1.
Regarding Claim 5. Ryan in view of Steinbrücker discloses The system of claim 1, wherein:
the surgical procedure is being performed by way of a computer-assisted surgical instrument;
the recognized object is the computer-assisted surgical instrument; and the synthetic element added to the original image for the augmented version of the original image is an anatomical model generated prior to the surgical procedure using a preoperative imaging modality.
(Ryan teaches “a mixed reality surgical navigation system (10) to be worn by a user (106) during a surgical procedure” (ABST). Figs. 31-33 show a surgical site at which a hip replacement surgery is being performed by an instrument. ¶129 teaches displaying a synthetic element (i.e. a virtual model) in a mixed reality view, and recites “FIG. 31 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 during registration of a spine with ultrasound. An anatomy marker 1300 is fixated to a vertebra adjacent to the operative site. An ultrasound probe 3104 which includes a plurality of fiducials 3106 defining a marker is provided. In one embodiment, the ultrasound probe 3104 is battery operated, cordless, and can communicate with the AR headset 3600 via radio. The software has geometric and other information necessary to be able to position and scale the 2D ultrasound image relative to the marker's 1300 position.” Further, ¶130 recites “FIG. 32 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 during registration of a spine with a stylus 1800. The anatomy marker 1300 is fixated to a vertebra adjacent to the operative site. A virtual model 3200 of the patient's vertebra generated from pre-operative imaging is displayed.”)
Regarding Claim 6. Ryan in view of Steinbrücker discloses The system of claim 1, wherein the recognized object is a computer-assisted surgical instrument being used to perform the surgical procedure, and the model is a three-dimensional (“3D”) computer-aided design (“CAD”) model of the computer-assisted surgical instrument. (Ryan, ¶172 reciting “The system may also be used to recognize surgical instruments and implants encountered during surgery. A database of CAD models of instruments and equipment to scale is held in memory.”)
Regarding Claim 7. Ryan in view of Steinbrücker discloses The system of claim 1, wherein the recognized object is held by a computer-assisted surgical instrument being used to perform the surgical procedure, and the model is generated by way of a 3D scan of the recognized object. (Ryan, Fig. 31; and ¶129 reciting “The posterior contour of the spinous process and the left and right mammillary processes can be matched to the same features of a CT generated 3D model of the vertebra to register and subsequently position the virtual model of the vertebra in a mixed reality view.”)
Regarding Claim 8. Ryan in view of Steinbrücker discloses The system of claim 1, wherein the processor is further configured to execute the instructions to provide the generated presentation data to the presentation system for use in presenting the augmented version of the original image. (Steinbrücker, ¶111 reciting “Described herein are methods and apparatus for creating and using a three-dimensional (3D) world reconstruction in an augmented reality (AR) . . system.”; and further, ¶356 recites “The realism with which AR and MR scenes are presented to a user may be enhanced by providing occlusion data to applications generating such scenes”. The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 16. Ryan in view of Steinbrücker discloses The method of claim 9, further comprising: presenting, based on the presentation data, an augmented image that includes the original image augmented with only a portion of the synthetic element. (Steinbrücker, ¶112 reciting “Occlusion processing identifies portions of a virtual object that should not be rendered for and/or displayed to a user because there is an object in the physical world blocking that user's view of the location where that virtual object is to be perceived by the user.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Claim 13, has similar limitations as of Claim(s) 5, therefore it is rejected under the same rationale as Claim(s) 5.
Claim 14, has similar limitations as of Claim(s) 6, therefore it is rejected under the same rationale as Claim(s) 6.
Claim 15, has similar limitations as of Claim(s) 7, therefore it is rejected under the same rationale as Claim(s) 7.
Claim(s) 3-4, 11-12, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ryan in view of Steinbrücker, and further in view of Lagmanson (US 20200090405 A1).
Regarding Claim 3, Ryan in view of Steinbrücker discloses The system of claim 1.
However, Ryan in view of Steinbrücker does not explicitly disclose the original image is a video image consisting of a sequence of image frames each depicting the recognized object as the recognized object moves with respect to other imagery depicted by the original image; and
the processor is further configured to execute the instructions to identify the recognized object within imagery depicted by the video image by initially recognizing the recognized object in a first image frame of the sequence of image frames, and tracking, based on the initial recognition, the recognized object from image frame to image frame as the recognized object moves with respect to the other imagery in later image frames of the sequence of image frames.
Lagmanson teaches detecting and tracking object in image frame in Fig. 4, step 420; and ¶57 recites “the method 400 may proceed to block 420 where an object to be positioned at the target coordinate of the location interest is detected and tracked in the image frame. . . The user may define the object such that the augmented reality object positioning controller 204 and/or 304 can recognize the object and track it by gathering feature points of the object, calculating a pose of the object, and tracking those feature points and pose throughout subsequent image frames.” In addition, ¶54 recites “the image frame may include . . . , a video of the physical environment 103”.
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the system (taught by Ryan in view of Steinbrücker) to recognize an object in a first image frame in a video and to track the recognized object in the subsequent image frames (taught by Lagmanson). The suggestions/motivations would have been “The systems and methods described herein provide more user friendly, less expensive, and/or less complex object positioning systems than compared to conventional surveying equipment and global positioning systems used to position objects such as geophysical sensors in a physical environment.” (¶5), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding Claim 4, Ryan in view of Steinbrücker and Lagmanson discloses The system of claim 1, wherein the processor is further configured to execute the instructions to:
access the kinematic data representative of the pose of the recognized object; and
identify the recognized object within imagery depicted by the original image based on the kinematic data.
(Lagmanson, ¶57 reciting “The user may define the object such that the augmented reality object positioning controller 204 and/or 304 can recognize the object and track it by gathering feature points of the object, calculating a pose of the object, and tracking those feature points and pose throughout subsequent image frames. In various embodiments, an object profile may be associated with the location of interest and the augmented reality object positioning controller 204 and/or 304 may be configured to perform object recognition such that when searching for the LOI 220 and/or 312 in the image frame, the augmented reality object positioning controller 204 and/or 304 is identifying objects within the image frame by capturing feature points of those objects and comparing the captured feature points to feature points of the object associated with the LOI 220 and/or 312 to determine whether that object associated with the LOI 220 and/or 312 is within the image frame.” The suggestions/motivations would have been the same as that of Claim 3 rejections.)
Claim 11, has similar limitations as of Claim(s) 3, therefore it is rejected under the same rationale as Claim(s) 3.
Claim 12, has similar limitations as of Claim(s) 4, therefore it is rejected under the same rationale as Claim(s) 4.
Claim 19, has similar limitations as of Claim(s) 3, therefore it is rejected under the same rationale as Claim(s) 3.
Claim 20, has similar limitations as of Claim(s) 4, therefore it is rejected under the same rationale as Claim(s) 4.
Allowable Subject Matter
Claims 2, 10, and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and overcome the double patenting rejections.
Regarding Claim 2, the closest art Ryan in view of Steinbrücker discloses The system of claim 1.
However, the closest art does not explicitly disclose wherein the associating of the model with the recognized object includes:
generating a depth map of imagery depicted by the original image, the depth map including first depth data for a depiction of the recognized object within the imagery and second depth data for a remainder of the imagery, the first depth data based on the model of the recognized object and denser than the second depth data; and
segmenting the original image to distinguish pixels of the original image that depict the recognized object from pixels of the original image that do not depict the recognized object by
identifying the pixels of the original image that depict the recognized object based on the first depth data; and
identifying the pixels of the original image that do not depict the recognized object based on the second depth data.
Claims 10 and 18 are each similar in scope to method claim 2 and also contain allowable subject matter.
Allowable Subject Matter
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YI WANG whose telephone number is (571)272-6022. The examiner can normally be reached 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YI WANG/Primary Examiner, Art Unit 2619
Prosecution Insights