DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS), submitted on 4/28/2025 and 10/8/2025, are being considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/forms/. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to http://www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
4. Claims 21, 25, and 35 are rejected on the ground of nonstatutory double patenting as being unpatentable over related claims of the U.S. Patent 11,682,165 B2. Although the conflicting claims are not identical, they are not patentably distinct from each other because the instant claims are similar to the claims in the US Patent 11,682,165 B2 to meet the limitations claimed in the co-pending applications. Table 1 shows comparisons between the instant claims and the US Patent 11,682,165 B2 claims.
Table 1: Comparison of claims in the instant Application 19/035,856 vs. the U.S. Patent 11,682,165 B2.
Instant Application 19/035,856
U.S. Patent 11,682,165 B2
21. A method of determining depth within a surgical scene, the method comprising:
capturing depth data of anatomy of a patient within the surgical scene undergoing a surgical procedure with a depth sensor; capturing image data of the surgical scene with a plurality of cameras different than the depth sensor;
receiving an input selecting a region of the anatomy for which higher resolution depth information is desired; processing the image data to generate additional depth data for the region of the anatomy for which higher resolution depth information is desired; and
merging the depth data of the anatomy from the depth sensor with the additional depth data to generate a merged depth map of the anatomy.
1. A method of determining the depth of a scene, the method comprising:
capturing depth data of the scene with a depth sensor;
capturing image data of the scene with a plurality of cameras;
generating a point cloud representative of the scene based on the depth data;
identifying a region of the point cloud;
generating depth data for the region based on the image data; and
merging the depth data for the region with the depth data from the depth sensor to generate a merged point cloud representative of the scene.
Nonn et al. (US Patent 11,682,165 B2) further meets the different claim limitations as follow:
receiving an input selecting a region of the anatomy for which higher resolution depth information is desired (In other embodiments, the user can select certain areas in which to improve the resolution) [Noun: col. 11, line 40-42].
25. The method of claim 21 wherein the input comprises a user selection of the region of the anatomy for which higher resolution depth information is desired.
4. The method of claim 1 wherein the depth data for the missing region has a greater resolution than the depth data captured with the depth sensor.
35. A system for imaging a surgical scene, comprising: multiple cameras arranged at different positions and orientations relative to the surgical scene and configured to capture image data of anatomy of a patient within the surgical scene undergoing a surgical procedure; a depth sensor configured to capture depth data of the anatomy; and a computing device communicatively coupled to the cameras and the depth sensor, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory, and wherein the computer-executable instructions,
when executed by the processor, cause the processor to:
receive the image data of the anatomy from the cameras;
receive the depth data of the anatomy from the depth sensor;receive an input selecting a region of the anatomy for which higher resolution depth information is desired;process the image data to generate additional depth data for the region of the anatomy for which higher resolution depth information is desired; and
merge the depth data of the anatomy from the depth sensor with the additional depth data to generate a merged depth map of the anatomy.
8. A system for imaging a scene, comprising:
multiple cameras arranged at different positions and orientations relative to the scene and configured to capture light field image data of the scene;
a depth sensor configured to capture depth data about a depth of the scene; and
a computing device communicatively coupled to the cameras and the depth sensor, wherein the computing device as a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory,
wherein the computer-executable instructions include instructions for
generating an output image of a portion of the scene, wherein the output image is from a perspective of a virtual camera having a field of view corresponding to the portion of the scene and wherein generating the output image comprises:
receiving the light field image data from the cameras·
receiving the depth data from the depth sensor '
generating a point cloud representative of the scene based on the depth data;
identifying a missing region of the point cloud within the field of view of the virtual camera in which the point cloud includes no data or sparse data, wherein identifying the missing region of the point cloud includes determining that the missing region of the point cloud has fewer than a predetermined threshold number of data points;
generating depth data for the missing region based on the light field image data;
merging the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the scene·
processing the light field image data and the merged point cloud to synthesize the output image of the portion scene from the perspective of the virtual camera· and
transmitting the output image to a display for display to a user.
Nonn et al. (US Patent 11,682,165 B2) further meets the different claim limitations as follow:
receive an input selecting a region of the anatomy for which higher resolution depth information is desired (In other embodiments, the user can select certain areas in which to improve the resolution) [Noun: col. 11, line 40-42].
process the image data to generate additional depth data for the region of the anatomy for which higher resolution depth information is desired (In one aspect of the present technology, the merged point cloud can have a greater accuracy and/or resolution than the point cloud generated from the depth data from the depth sensor alone.) [Noun: col. 2, line 52-55].
5. Claims 21, 22, 25, 33-38 are rejected on the ground of nonstatutory double patenting as being unpatentable over related claims of the U.S. Patent 12243162 B2. Although the conflicting claims are not identical, they are not patentably distinct from each other because the instant claims are similar to the claims in the US Patent 12243162 B2 to meet the limitations claimed in the co-pending applications. Table 1 shows comparisons between the instant claims and the US Patent 12243162 B2 claims.
Table 1: Comparison of claims in the instant Application 19/035,856 vs. the U.S. Patent 12243162 B2.
Instant Application 19/035,856
U.S. Patent 12243162 B2
21. A method of determining depth within a surgical scene, the method comprising:
capturing depth data of anatomy of a patient within the surgical scene undergoing a surgical procedure with a depth sensor; capturing image data of the surgical scene with a plurality of cameras different than the depth sensor;
receiving an input selecting a region of the anatomy for which higher resolution depth information is desired; processing the image data to generate additional depth data for the region of the anatomy for which higher resolution depth information is desired; and
merging the depth data of the anatomy from the depth sensor with the additional depth data to generate a merged depth map of the anatomy.
1. A method of determining a depth within a surgical scene, the method comprising:
capturing depth data of anatomy of a patient within the surgical scene undergoing a surgical procedure with a depth sensor;
capturing image data of the anatomy with a plurality of cameras;
generating a point cloud representative of the anatomy based on the depth data from the depth sensor;
identifying a missing region of the point cloud in which the point cloud includes no data or sparse data, wherein the missing region corresponds to a portion of the anatomy that is occluded from the depth sensor;
determining at least one depth value from the point cloud for an area adjacent to the missing region;
processing the image data with a depth processing algorithm to generate depth data for the missing region,
wherein processing the image data with the depth
processing algorithm includes limiting the depth processing algorithm to determine the depth data for the missing region in a depth range urrounding the at least one depth value; and
merging the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the anatomy.
Nonn et al. (US Patent 12243162 B2) further meets the different claim limitations as follow:
receiving an input selecting a region of the anatomy for which higher resolution depth information is desired (In other embodiments, the user can select certain areas in which to improve the resolution) [Noun: col. 11, line 42-44].
22. The method of claim 21 wherein the image data is light field image data.
8. The method of claim 1 wherein the image data is light field image data.
25. The method of claim 21 wherein the input comprises a user selection of the region of the anatomy for which higher resolution depth information is desired.
4. The method of claim 1 wherein the depth data for the missing region has a greater resolution than the depth data captured with the depth sensor.
33. The method of claim 21 wherein the cameras and the depth sensor are rigidly mounted to a common frame and fixed in position relative to one another.
6. The method of claim 1 wherein the cameras and the depth sensor are rigidly mounted to a common frame and fixed in position relative to one another.
34. The method of claim 33 wherein the cameras are RGB cameras.
7. The method of claim 6 wherein the cameras are RGB cameras.
35. A system for imaging a surgical scene, comprising: multiple cameras arranged at different positions and orientations relative to the surgical scene and configured to capture image data of anatomy of a patient within the surgical scene undergoing a surgical procedure; a depth sensor configured to capture depth data of the anatomy; and a computing device communicatively coupled to the cameras and the depth sensor, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory, and wherein the computer-executable instructions,
when executed by the processor, cause the processor to:receive the depth data of the anatomy from the depth sensor;receive the image data of the anatomy from the cameras; receive an input selecting a region of the anatomy for which higher resolution depth information is desired;process the image data to generate additional depth data for the region of the anatomy for which higher resolution depth information is desired; and
merge the depth data of the anatomy from the depth sensor with the additional depth data to generate a merged depth map of the anatomy.
9. A system for imaging a surgical scene, comprising:
multiple cameras arranged at different positions and orientations relative to the surgical scene and configured to capture image data of anatomy of a patient within the surgical scene undergoing a surgical procedure;
a depth sensor configured to capture depth data of the anatomy; and
a computing device communicatively coupled to the cameras and the depth sensor, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory,
and wherein the computer-executable instructions,
when executed by the processor, cause the processor to:
receive the depth data of the anatomy from the depth sensor;
receive the image data of the anatomy from the cameras;
generate a point cloud representative of the anatomy based on the depth data from the depth sensor;
identify a missing region of the point cloud in which the point cloud includes no data or sparse data,
wherein the missing region corresponds to a portion of the anatomy that is occluded from the depth sensor;
determine at least one depth value from the point cloud for an area adjacent to the missing region;
process the image data with a depth processing algorithm to generate depth data for the missing region,
wherein the computer-executable instructions, when executed by the processor, further cause the processor to limit the depth processing algorithm to determine the depth data for the missing region in a depth range surrounding the at least one depth value; and
merge the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the anatomy.
Nonn et al. (US Patent 12243162 B2) further meets the different claim limitations as follow:
receive an input selecting a region of the anatomy for which higher resolution depth information is desired (In other embodiments, the user can select certain areas in which to improve the resolution) [Noun: col. 11, line 40-42].
36. The system of claim 35 wherein the cameras and the depth sensor are rigidly mounted to a common frame and fixed in position relative to one another.
14. The system of claim 9 wherein the cameras and the depth sensor are rigidly mounted to a common frame and fixed in position relative to one another.
37. The system of claim 36 wherein the cameras are RGB cameras.
15. The system of claim 14 wherein the cameras are RGB cameras.
38. The system of claim 35 wherein the input comprises a user selection of the region of the anatomy for which higher resolution depth information is desired.
12. The system of claim 9 wherein the depth data for the missing region has a greater resolution than the depth data captured with the depth sensor.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102(e), (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 21-23 and 25-39 are rejected under 35 U.S.C. 103 as being unpatentable over Hong (US Patent 10,121,064 B2), (“Hong”), in view of Srimohanarajah et al. (US Patent 11,045,257 B2), (“Srimohanarajah”), in view of McDowall et al. (US Patent 10,832,429 B2), (“McDowall”).
Regarding claim 21, Hong meets the claim limitations, as follows:
A method (Systems and methods) [Hong: col. 2, line 12] of determining a depth (The behavioral classification system 100 includes a imaging system 102 that is capable of capturing image data including depth information. Depth information typically refers to a measurement of a distance from a reference viewpoint to one point or many points in a scene) [Hong: col. 10, line 23-28] of a surgical scene, the method (methods) [Hong: col. 2, line 12] comprising: capturing depth data (capturing image data including depth information) [Hong: col. 10, line 25-26] of anatomy of a patient within the surgical scene undergoing a surgical procedure with a depth sensor (depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]; capturing image data of the surgical scene ((a camera is used to acquire images of one or more subjects) [Hong: col. 10, line 29-30]; (identifies (508) the subjects and their locations within the imaged scene) [Hong: col. 17, line 58-59]) with a plurality of cameras ((Many of the behavioral classification systems described above synchronize image data captured by one or more cameras and one or more depth sensors) [Hong: col. 30, line 61-63]; (Other depth sensors could be utilized to obtain depth information such as, but not limited to, a plurality of cameras configured to capture information in color channels including the near-IR color channel in a multiview stereo configuration in combination with an illumination source configured to project texture onto the scene.) [Hong: col. 16, line 30-36]) different than the depth sensor (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]; receiving (The computer system includes a processor 106 that receives) [Hong: col. 11, line 35] an input selecting a region ((an interactive user interface) [Hong: col. 12, line 36]; (inputs to behavior classification processes) [Hong: col. 11, line 18]) of the anatomy for which higher resolution depth information is desired; processing the image data (The captured image data that includes depth information is then analyzed via an image processing pipeline) [Hong: col. 2, line 4-6] to generate additional depth data ((i.e. Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification of social behaviors.) [Hong: col. 1, line 60-65]; (i.e. As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]) for the region of the anatomy for which higher resolution depth information is desired ((During image data capture, data is acquired synchronously by all three devices to produce simultaneous depth information and top and side view grayscale videos) [Hong: col. 15, line 52-54; Figs. 3D-9]; (video recordings from the top view camera are projected into the viewpoint of the depth information captured by the depth sensor to create a common coordinate framework) [Hong: col. 17, line 25-28]; (Compute the average depth ZiR(t) within a square region) [Hong: col. 21, line 46]); and merging the depth data of the anatomy from the depth sensor ((combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67]; (Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification) [Hong: col. 1, line 60-65]; (classification can be performed based upon raw image data, detected pose and raw 3D trajectory information, and/or any combination of raw data, pose data, trajectory data, and/or parameters appropriate to the requirements of a specific application) [Hong: col. 13, line 44-48]; (the average depth ZiR(t) within a square region) [Hong: col. 21, line 46]; (As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]) with the additional depth data (a behavioral classification system in accordance with an embodiment of the invention was constructed that records behavior using synchronized conventional video cameras and a time-of-flight depth sensor) [Hong: col. 15, line 36-38; Figs. 3A-3D] to generate a merged depth map of the anatomy ((Depth information can be obtained using any of a variety of depth sensors including (but not limited to) a time of flight depth sensor, a structured illumination depth sensor, a Light Detection and Ranging (LIDAR) sensor, a Sound Navigation and Ranging (SONAR) sensor, an array of two or more conventional cameras in a multiview stereo configuration, and/or an array of two or more conventional cameras in a multiview stereo configuration in combination with an illumination source that projects texture onto a scene to assist with parallax depth information recovery on otherwise textureless surfaces) [Hong: col. 10, line 44-54; Figs. 3A-9]; (In the illustrated experimental apparatus, the top view camera 304 and the depth sensor 306 are mounted as close together as possible (see FIG. 3D) to limit occlusions (pixels in the images captured by the top view camera for which depth information is not available due to the occlusion of that pixel location in the field of view of the depth camera). In the illustrated embodiment, the depth sensor is a time-of-flight depth sensor that includes an IR illumination source 320 and an IR camera 322 and detects contours of objects in the depth or z-plane by measuring the time-of flight of an infrared light signal generated by the IR illumination source 320 between the depth sensor and object surfaces for each point of the depth image generated by the time-of-flight depth sensor, in a manner analogous to SONAR) [Hong: col. 16, line 13-27; Figs. 3A-3D]; (As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]).
Hong does not explicitly disclose the following claim limitations (Emphasis added).
a surgical scene,
anatomy of a patient within the surgical scene undergoing a surgical procedure;
a region of the anatomy;
higher resolution depth information is desired.
However, in the same field of endeavor Srimohanarajah further discloses the deficient claim limitations as follows:
a surgical scene (the operation room) [Srimohanarajah: col. 2, line 35; Fig. 2]; anatomy of a patient within the surgical scene (CT is often used to visualize boney structures and blood vessels) [Srimohanarajah: col. 1, line 38-39; Fig. 2] undergoing a surgical procedure (on a patient's surgical position) [Srimohanarajah: col. 2, line 63; Fig. 2]; (involved in a surgical procedure) [Srimohanarajah: col. 3, line 56-57; Figs. 4A-B];
a region of the anatomy (perform a surgical procedure involving tumor resection in which the residual tumor remaining after is minimized, while also minimizing the) [Srimohanarajah: col. 5, line 7-9; Fig. 2];
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
higher resolution depth information is desired.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations as follows:
higher resolution depth information is desired ((The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The
controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20] ; (The sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260. Thus, the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor. Here, the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution) [McDowall: col. 28, line 39-49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Regarding claim 22, Hong meets the claim limitations as set forth in claim 21. Hong further meets the claim limitations as follow.
wherein the image data (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13] is light field image data.
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
wherein the image data is light field image data.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations, as follows:
wherein the image data is light field image data (the images captured as samples of a light field) [McDowall: col. 52, line 49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Regarding claim 23, Hong meets the claim limitations as set forth in claim 21. Hong further meets the claim limitations as follow.
wherein the merged depth map (combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67] is a point cloud ((point clouds or meshes generated using image data) [Hong: col. 20, line 23-24]; (image data including depth information) [Hong: col. 10, line 25-26]).
Regarding claims 25 and 38, Hong meets the claim limitations as set forth in claims 21 and 35. Hong further meets the claim limitations as follow.
wherein the input comprises a user selection of the region of the anatomy ((an interactive user interface) [Hong: col. 12, line 36]; (inputs to behavior classification processes) [Hong: col. 11, line 18]) for which higher resolution depth information is desired.
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
higher resolution depth information is desired.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations, as follows:
higher resolution depth information is desired ((The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The
controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20] ; (The sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260. Thus, the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor. Here, the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution) [McDowall: col. 28, line 39-49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Regarding claims 26 and 39, Hong meets the claim limitations as set forth in claims 21 and 35. Hong further meets the claim limitations as follow.
wherein the input comprises ((an interactive user interface) [Hong: col. 12, line 36]; (inputs to behavior classification processes) [Hong: col. 11, line 18]) comprises tracking data of one or more instruments within the surgical scene (directs the processor to perform a number of image processing applications designed to track one or more subjects in the captured image data 112 in 3D) [Hong: col. 11, line 18].
In the same field of endeavor Srimohanarajah further discloses the claim limitations as follows:
tracking data of one or more instruments within the surgical scene (Tracking of instruments relative to the patient and the associated imaging data is also often achieved by way of external hardware systems such as mechanical arms, or radiofrequency or optical tracking devices. As a set, these devices are commonly referred to as surgical navigation systems) [Srimohanarajah: col. 1, line 57-62].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, McDowall with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong, McDowall, with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Regarding claim 27, Hong meets the claim limitations as set forth in claim 21. Hong further meets the claim limitations as follow.
registering the merged depth map (combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67] with medical scan data of the anatomy.
Hong and McDowall do not explicitly disclose the following claim limitations (Emphasis added).
registering the merged depth map with medical scan data of the anatomy.
In the same field of endeavor Srimohanarajah further discloses the claim limitations as follows:
registering the merged depth map with medical scan data of the anatomy ((i.e. registering the tracking system to create the single unified virtual coordinate space for the 3D scan data, the medical image data) [Srimohanarajah: col. 16, line 21-23]; (i.e. Using a dense point cloud provided by the 3D scanner 309, this point cloud may be mapped to the extracted surface of the MR/CT volumetric scan data ( e.g., the pre-op image data 354) to register the patient's physical position to the volumetric data. The tracking system 321 (e.g., part of the navigation system 200) has no reference to the point cloud data. Therefore a tool may be provided that is visible to both the tracking system 321 and the 3D scanner 309. A transformation between the tracking system's camera space and the 3D scanner space may be identified so that the point cloud provided by the 3D scanner 309 and the tracking system 321 can be registered to the patient space.) [Srimohanarajah: col. 10, line 53-65; Fig. 9]; (i.e. the method 900 generates and receives 3D scan data from the 3D scanner 309 that is representative of a 3D scan of at least a portion of the patient 202.) [Srimohanarajah: col. 13, line 24-26]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, McDowall with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong, McDowall, with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Regarding claims 28 and 31, Hong and McDowall meet the claim limitations as set forth in claims 21 and 30. Hong and McDowall do not explicitly disclose the following claim limitations (Emphasis added).
the medical scan data comprises computed tomography (CT) scan data.
In the same field of endeavor Srimohanarajah further discloses the claim limitations as follows:
the medical scan data ((i.e. registering the tracking system to create the single unified virtual coordinate space for the 3D scan data, the medical image data) [Srimohanarajah: col. 16, line 21-23]; ; (i.e. the method 900 generates and receives 3D scan data from the 3D scanner 309 that is representative of a 3D scan of at least a portion of the patient 202.) [Srimohanarajah: col. 13, line 24-26]) comprises computed tomography (CT) scan data (i.e. Using a dense point cloud provided by the 3D scanner 309, this point cloud may be mapped to the extracted surface of the MR/CT volumetric scan data ( e.g., the pre-op image data 354) to register the patient's physical position to the volumetric data. The tracking system 321 (e.g., part of the navigation system 200) has no reference to the point cloud data. Therefore a tool may be provided that is visible to both the tracking system 321 and the 3D scanner 309. A transformation between the tracking system's camera space and the 3D scanner space may be identified so that the point cloud provided by the 3D scanner 309 and the tracking system 321 can be registered to the patient space.) [Srimohanarajah: col. 10, line 53-65; Fig. 9].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, McDowall with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong, McDowall, with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Regarding claim 29, Hong meets the claim limitations as set forth in claim 21. Hong further meets the claim limitations as follow.
generating a point cloud representative of the anatomy ((i.e. point clouds or meshes generated using image data) [Hong: col. 20, line 23-24]; (i.e. image data including depth information) [Hong: col. 10, line 25-26]) based on the depth data from the depth sensor ((i.e. depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]; (i.e. image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]); and registering the point cloud ((i.e. point clouds or meshes generated using image data) [Hong: col. 20, line 23-24]; (i.e. image data including depth information) [Hong: col. 10, line 25-26]) with medical scan data of the anatomy.
Hong and McDowall do not explicitly disclose the following claim limitations (Emphasis added).
registering the point cloud with medical scan data of the anatomy.
In the same field of endeavor Srimohanarajah further discloses the claim limitations as follows:
registering the point cloud with medical scan data of the anatomy ((i.e. registering the tracking system to create the single unified virtual coordinate space for the 3D scan data, the medical image data) [Srimohanarajah: col. 16, line 21-23]; (i.e. Using a dense point cloud provided by the 3D scanner 309, this point cloud may be mapped to the extracted surface of the MR/CT volumetric scan data ( e.g., the pre-op image data 354) to register the patient's physical position to the volumetric data. The tracking system 321 (e.g., part of the navigation system 200) has no reference to the point cloud data. Therefore a tool may be provided that is visible to both the tracking system 321 and the 3D scanner 309. A transformation between the tracking system's camera space and the 3D scanner space may be identified so that the point cloud provided by the 3D scanner 309 and the tracking system 321 can be registered to the patient space.) [Srimohanarajah: col. 10, line 53-65; Fig. 9]; (i.e. the method 900 generates and receives 3D scan data from the 3D scanner 309 that is representative of a 3D scan of at least a portion of the patient 202.) [Srimohanarajah: col. 13, line 24-26]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, McDowall with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong, McDowall, with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Regarding claim 30, Hong meets the claim limitations as set forth in claim 30. Hong further meets the claim limitations as follow.
registering the merged depth map (combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67] with medical scan data of the anatomy.
wherein processing the image data comprises processing the image data (i.e. The captured image data that includes depth information is then analyzed via an image processing pipeline) [Hong: col. 2, line 4-6] with a depth processing algorithm to generate the additional depth data ((i.e. Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification of social behaviors.) [Hong: col. 1, line 60-65]; (i.e. combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67]), and wherein processing the image data with the depth processing algorithm (i.e. Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification of social behaviors.) [Hong: col. 1, line 60-65] includes limiting the depth processing algorithm to determine the additional depth data in a depth range surrounding (i.e. As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16] the registered medical scan data in the region for which higher resolution depth information is desired.
Hong does not explicitly disclose the following claim limitations (Emphasis added).
the registered medical scan data in the region for which higher resolution depth information is desired.
In the same field of endeavor Srimohanarajah further discloses the claim limitations as follows:
the registered medical scan data in the region ((i.e. registering the tracking system to create the single unified virtual coordinate space for the 3D scan data, the medical image data) [Srimohanarajah: col. 16, line 21-23]; (i.e. Using a dense point cloud provided by the 3D scanner 309, this point cloud may be mapped to the extracted surface of the MR/CT volumetric scan data ( e.g., the pre-op image data 354) to register the patient's physical position to the volumetric data. The tracking system 321 (e.g., part of the navigation system 200) has no reference to the point cloud data. Therefore a tool may be provided that is visible to both the tracking system 321 and the 3D scanner 309. A transformation between the tracking system's camera space and the 3D scanner space may be identified so that the point cloud provided by the 3D scanner 309 and the tracking system 321 can be registered to the patient space.) [Srimohanarajah: col. 10, line 53-65; Fig. 9]; (i.e. the method 900 generates and receives 3D scan data from the 3D scanner 309 that is representative of a 3D scan of at least a portion of the patient 202.) [Srimohanarajah: col. 13, line 24-26]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, McDowall with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong, McDowall, with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
higher resolution depth information is desired.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations as follows:
higher resolution depth information is desired ((The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The
controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20] ; (The sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260. Thus, the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor. Here, the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution) [McDowall: col. 28, line 39-49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Regarding claim 32, Hong meets the claim limitations as set forth in claim 21. Hong further meets the claim limitations as follow.
wherein the additional depth data (i.e. pixels in the images captured by the top view camera for which depth information is not available due to the occlusion of that pixel location in the field of view of the depth camera) [Hong: col. 16, line 16-19] has a higher resolution than the depth data from the depth sensor (i.e. FIG. 7A shows raw depth image data acquired by a depth sensor) [Hong: col. 7, line 4-5; Fig. 7A].
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
a higher resolution.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations as follows:
a higher resolution depth ((The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20]; (The sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than
is possible if only an image from a single image capture sensor is processed by central controller 260. Thus, the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor. Here, the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution) [McDowall: col. 28, line 39-49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Regarding claims 33 and 36, Hong meets the claim limitations as set forth in claims 21 and 35. Hong further meets the claim limitations as follow.
wherein the cameras and the depth sensor are rigidly mounted to a common frame and fixed in position relative to one another (i.e. In the illustrated experimental apparatus, the top view camera 304 and the depth sensor 306 are mounted as close together as possible (see FIG. 3D) to limit occlusions (i.e. pixels in the images captured by the top view camera for which depth information is not available due to the occlusion of that pixel location in the field of view of the depth camera)) [Hong: col. 16, line 13-19; Figs. 3C-D].
Regarding claims 34 and 37, Hong meets the claim limitations as set forth in claims 21 and 35. Hong further meets the claim limitations as follow.
wherein the cameras are RGB cameras ((i.e. a Bayer camera) [Hong: col. 33, line 47]; (i.e. image data take the form of so called RGB-D data (i.e. Red, Green, Blue, and Depth image data)) [Hong: col. 11, line 7-8]).
Regarding claim 35, Hong meets the claim limitations, as follows:
A system (Systems and methods) [Hong: col. 2, line 12] for imaging a surgical scene (The behavioral classification system 100 includes a imaging system 102 that is capable of capturing image data including depth information. Depth information typically refers to a measurement of a distance from a reference viewpoint to one point or many points in a scene) [Hong: col. 10, line 23-28], comprising: multiple cameras (( Many of the behavioral classification systems described above synchronize image data captured by one or more cameras and one or more depth sensors) [Hong: col. 30, line 61-63]; (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]; (a plurality of cameras configured to capture information in color channels including the near-IR color channel in a multiview stereo configuration in combination with an illumination source configured to project texture onto the scene.) [Hong: col. 16, line 32-36]) arranged at different positions and orientations relative to the surgical scene ((a plurality of cameras in a multiview stereo configuration) [Hong: col. 4, line 5; Figs. 3A-8F]; (an array of two or more conventional cameras in a multiview stereo configuration, and/or an array of two or more conventional cameras in a multiview stereo configuration in combination with an illumination source that projects texture onto a scene to assist with parallax depth information recovery on otherwise textureless surfaces) [Hong: col. 10, line 50-54; Figs. 3A-9]; (The behavioral classification system illustrated in FIGS. 3A-3D can be used to track animal trajectories and orientations in 3D in the context of an animal's home cage and detect specific social behaviors, including attack, mounting and close investigation in different orientations (head-to-head, head-to-tail, head-to-side, etc).) [Hong: col. 16, line 40-45; Figs. 3A-8F]; (position and pose information is passed through a set of feature extractors to obtain a low-dimensional representation from which machine learning algorithms can be used to train classifiers to detect specific behaviors. In other embodiments, the raw position and pose information can be passed directly to the classifier) [Hong: col. 13, line 8-13]) and configured to capture image data ((a camera is used to acquire images of one or more subjects) [Hong: col. 10, line 29-30]; (identifies (508) the subjects and their locations within the imaged scene) [Hong: col. 17, line 58-59]) of anatomy of a patient within the surgical scene undergoing a surgical procedure ((Many of the behavioral classification systems described above synchronize image data captured by one or more cameras and one or more depth sensors) [Hong: col. 30, line 61-63]; (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]; (depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]); a depth sensor (depth sensor) [Hong: col. 10, line 30] configured to capture depth data of the anatomy ((depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]; (capturing image data including depth information) [Hong: col. 10, line 25-26]); and a computing device (a microprocessor) [Hong: col. 31, line 38] communicatively coupled to the cameras and the depth sensor ((a plurality of 3D imaging systems and a behavioral classification computer system including at least one memory and at least one microprocessor directed by at least a classification application stored in the at least one memory to: control the plurality of 3D imaging systems to each acquire a sequence of frames of image data including depth information; and store at least a portion of each of the sequences of frames of image data including depth information in the at least one memory) [Hong: col. 5, line 4-14; Fig. 1]; (the 3D imaging system is selected from the group consisting of: a time of flight depth sensor and at least one camera; a structured light depth sensor and at least one camera; a LIDAR depth sensor and at least one camera; a SONAR depth sensor and at least one camera; a plurality of cameras in a multiview stereo configuration; and a plurality of cameras in multiview stereo configuration and an illumination source that projects texture) [Hong: col. 3, line 66 – col. 4, line 8; Fig. 1]), wherein the computing device (a microprocessor) [Hong: col. 31, line 38] has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory (Machine readable instructions stored in memory 108 can be used to control the operations performed by the processor 106) [Hong: col. 11, line 63-65], and wherein the computer-executable instructions, when executed by the processor, cause the processor to (Machine readable instructions stored in memory 108 can be used to control the operations performed by the processor 106) [Hong: col. 11, line 63-65]:receive the depth data (capturing image data including depth information) [Hong: col. 10, line 25-26] of anatomy from the depth sensor (depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]; receive image data of the anatomy ((a camera is used to acquire images of one or more subjects) [Hong: col. 10, line 29-30]; (identifies (508) the subjects and their locations within the imaged scene) [Hong: col. 17, line 58-59]) from the cameras ((Many of the behavioral classification systems described above synchronize image data captured by one or more cameras and one or more depth sensors) [Hong: col. 30, line 61-63]; (Other depth sensors could be utilized to obtain depth information such as, but not limited to, a plurality of cameras configured to capture information in color channels including the near-IR color channel in a multiview stereo configuration in combination with an illumination source configured to project texture onto the scene.) [Hong: col. 16, line 30-36]) different than the depth sensor (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]; receive (The computer system includes a processor 106 that receives) [Hong: col. 11, line 35] an input selecting a region ((an interactive user interface) [Hong: col. 12, line 36]; (inputs to behavior classification processes) [Hong: col. 11, line 18]) of the anatomy for which higher resolution depth information is desired; process the image data (The captured image data that includes depth information is then analyzed via an image processing pipeline) [Hong: col. 2, line 4-6] to generate additional depth data ((i.e. Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification of social behaviors.) [Hong: col. 1, line 60-65]; (i.e. As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]) for the region of the anatomy for which higher resolution depth information is desired ((During image data capture, data is acquired synchronously by all three devices to produce simultaneous depth information and top and side view grayscale videos) [Hong: col. 15, line 52-54; Figs. 3D-9]; (video recordings from the top view camera are projected into the viewpoint of the depth information captured by the depth sensor to create a common coordinate framework) [Hong: col. 17, line 25-28]; (Compute the average depth ZiR(t) within a square region) [Hong: col. 21, line 46]); and merge the depth data of the anatomy from the depth sensor ((combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67]; (Systems and methods in accordance with various embodiments of the invention utilize integrated hardware and software systems that combine video tracking, depth sensing, machine vision and machine learning, for automatic detection and quantification) [Hong: col. 1, line 60-65]; (classification can be performed based upon raw image data, detected pose and raw 3D trajectory information, and/or any combination of raw data, pose data, trajectory data, and/or parameters appropriate to the requirements of a specific application) [Hong: col. 13, line 44-48]; (the average depth ZiR(t) within a square region) [Hong: col. 21, line 46]; (As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]) with the additional depth data (a behavioral classification system in accordance with an embodiment of the invention was constructed that records behavior using synchronized conventional video cameras and a time-of-flight depth sensor) [Hong: col. 15, line 36-38; Figs. 3A-3D] to generate a merged depth map of the anatomy ((Depth information can be obtained using any of a variety of depth sensors including (but not limited to) a time of flight depth sensor, a structured illumination depth sensor, a Light Detection and Ranging (LIDAR) sensor, a Sound Navigation and Ranging (SONAR) sensor, an array of two or more conventional cameras in a multiview stereo configuration, and/or an array of two or more conventional cameras in a multiview stereo configuration in combination with an illumination source that projects texture onto a scene to assist with parallax depth information recovery on otherwise textureless surfaces) [Hong: col. 10, line 44-54; Figs. 3A-9]; (In the illustrated experimental apparatus, the top view camera 304 and the depth sensor 306 are mounted as close together as possible (see FIG. 3D) to limit occlusions (i.e. pixels in the images captured by the top view camera for which depth information is not available due to the occlusion of that pixel location in the field of view of the depth camera). In the illustrated embodiment, the depth sensor is a time-of-flight depth sensor that includes an IR illumination source 320 and an IR camera 322 and detects contours of objects in the depth or z-plane by measuring the time-of flight of an infrared light signal generated by the IR illumination source 320 between the depth sensor and object surfaces for each point of the depth image generated by the time-of-flight depth sensor, in a manner analogous to SONAR) [Hong: col. 16, line 13-27; Figs. 3A-3D]; (As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]).
Hong does not explicitly disclose the following claim limitations (Emphasis added).
a surgical scene,
anatomy of a patient within the surgical scene undergoing a surgical procedure;
a region of the anatomy;
higher resolution depth information is desired.
However, in the same field of endeavor Srimohanarajah further discloses the deficient claim limitations as follows:
a surgical scene (the operation room) [Srimohanarajah: col. 2, line 35; Fig. 2]; anatomy of a patient within the surgical scene (CT is often used to visualize boney structures and blood vessels) [Srimohanarajah: col. 1, line 38-39; Fig. 2] undergoing a surgical procedure (on a patient's surgical position) [Srimohanarajah: col. 2, line 63; Fig. 2]; (involved in a surgical procedure) [Srimohanarajah: col. 3, line 56-57; Figs. 4A-B];
a region of the anatomy (perform a surgical procedure involving tumor resection in which the residual tumor remaining after is minimized, while also minimizing the) [Srimohanarajah: col. 5, line 7-9; Fig. 2];
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong with Srimohanarajah to install the 3D scanner system and a camera of the medical navigation system into the imaging system.
Therefore, the combination of Hong with Srimohanarajah will enable the system to generate 3D scanning medical data of the patient [Srimohanarajah: col. 3, line 27-39; Fig. 2; col. 10, line 53-65; col. 16, line 21-23; Fig. 9].
Hong and Srimohanarajah do not explicitly disclose the following claim limitations (Emphasis added).
higher resolution depth information is desired.
However, in the same field of endeavor McDowall further discloses the deficient claim limitations as follows:
higher resolution depth information is desired ((The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The
controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20] ; (The sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260. Thus, the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor. Here, the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution) [McDowall: col. 28, line 39-49]).
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong and Srimohanarajah with McDowall to implement McDowall’s method.
Therefore, the combination of Hong and Srimohanarajah with McDowall will enable the system to improve spatial resolution and sharpness relative to an image captured by a color image capture sensor [McDowall: col. 6, line 11-13].
Claims 24, and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Hong (US Patent 10,121,064 B2), (“Hong”), in view of Srimohanarajah et al. (US Patent 11,045,257 B2), (“Srimohanarajah”), in view of McDowall et al. (US Patent 10,832,429 B2), (“McDowall”), in view of Claret et al. (US Patent 10,832,429 B2), (“Claret”).
Regarding claims 24 and 40, Hong, Srimohanarajah, and McDowall meet the claim limitations as set forth in claims 21 and 35. Hong, Srimohanarajah, and McDowall further meet the claim limitations as follow.
generating a point cloud representative of the anatomy ((point clouds or meshes generated using image data) [Hong: col. 20, line 23-24]; (image data including depth information) [Hong: col. 10, line 25-26]) based on the depth data from the depth sensor ((depth sensor is used to acquire depth information) [Hong: col. 10, line 30-31]; (image data captured by one or more conventional video cameras, as well as depth sensors) [Hong: col. 10, line 11-13]);
generating a point cloud representative of the anatomy (point clouds or meshes generated using image data) [Hong: col. 20, line 23-24] based on the depth data from the depth sensor (combines information from the video and depth camera recordings) [Hong: col. 16, line 66-67]; determining at least one depth value from the point cloud for an area adjacent to the region of the anatomy (generated of the number of frames embedded within each of 31 2D regions delineated by running the watershed algorithm on the entire dataset) [Hong: col. 26, line 15-18] for which higher resolution depth information is desired ((video recordings from the top view camera are projected into the viewpoint of the depth information captured by the depth sensor to create a common coordinate framework) [Hong: col. 17, line 25-28]; (The controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor) [McDowall: col. 6, line 11-13; Figs. 8A-D]; (The controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor) [McDowall: col. 5, line 14-20]); andprocessing the image data with a depth processing algorithm to generate the additional depth data ((As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]; (a behavioral classification system in accordance with an embodiment of the invention was constructed that records behavior using synchronized conventional video cameras and a time-of-flight depth sensor) [Hong: col. 15, line 36-38; Figs. 3A-3D]), wherein processing the image data with the depth processing algorithm includes limiting the depth processing algorithm (position and pose information is passed through a set of feature extractors to obtain a low-dimensional representation from which machine learning algorithms can be used to train classifiers to detect specific behaviors. In other embodiments, the raw position and pose information can be passed directly to the classifier) [Hong: col. 13, line 8-13]) to determine the additional depth data in a depth range surrounding ((As is discussed further below, the use of depth information as an additional modality in combination with conventional video data can significantly enhance the accuracy and robustness of automated behavioral classification processes) [Hong: col. 11, line 13-16]; (a behavioral classification system in accordance with an embodiment of the invention was constructed that records behavior using synchronized conventional video cameras and a time-of-flight depth sensor) [Hong: col. 15, line 36-38; Figs. 3A-3D]) the at least one depth value.
Hong, Srimohanarajah, and McDowan do not explicitly disclose the following claim limitations (Emphasis added).
at least one depth value,
However, in the same field of endeavor Claret further discloses the claim limitations and the deficient claim limitations, as follows:
at least one depth value (i.e. The method may comprise an additional stage to generate a sparse depth map considering the slope of the epipolar lines obtained in the previous stage. The sparse depth map is obtained by assigning depth values (dz) of objects in the real world to the edges calculated before (dx dy)) [Claret: col. 24, line 42-46]); and merging the depth data for the missing region ((i.e. a sparse depth map showing three objects at different depths) [Claret: col. 13, line 14-15; Fig. 8]; (i.e. The method may comprise an additional stage to generate a sparse depth map considering the slope of the epipolar lines obtained in the previous stage. The sparse depth map is obtained by assigning depth values (dz) of objects in the real world to the edges calculated before (dx dy).) [Claret: col. 24, line 42-46])
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Hong, Srimohanarajah, and McDowan with Claret to use sparse depth map representation in the application.
Therefore, the combination of Hong, Srimohanarajah, and McDowan with Claret will enable the system to show objects in different depths [Claret: col. 13, line 14-15; col. 24, line 42-46; Fig. 8].
Reference Notice
Additional prior arts, included in the Notice of Reference Cited, made of record and not relied upon is considered pertinent to applicant's disclosure.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Philip Dang whose telephone number is (408) 918-7529. The examiner can normally be reached on Monday-Thursday between 8:30 am - 5:00 pm (PST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sath Perungavoor can be reached on 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000./Philip P. Dang/Primary Examiner, Art Unit 2488
Prosecution Insights