DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments filed 08/19/2026 have been considered but are moot in view of a new ground of rejections.
Further, with respect to Applicant’s arguments on 35 U.S.C. 101 rejections of claim 15 are not persuasive.
Specifically, as amended, claim 15 recites “a computer-readable medium”. However, in the state of the art, transitory signals are commonplace as a medium for transmitting computer instruction and thus, in the absence of any evidence to the contrary and give the broadest reasonable interpretation, the scope of a "computer readable medium" covers a signal per se."
A “magnetic, optical, electromagnetic, infrared, … or propagation medium” is neither a process nor a product, (i.e., a tangible “thing”) and therefore does not fall within one of the four statutory classes of § 101. Rather, a “magnetic, optical, electromagnetic, infrared, … or propagation medium” is a form of energy, in the absence of any physical structure or tangible material.
Therefore, claim 15 stands rejected under 35 U.S.C. 101.
Further, Applicant’s arguments on page 6 with respect to Sugimoto’s teachings regarding a depth map are not persuasive.
Specifically, on page 6, Applicant argues that,
“The depth information in Sugimoto is a relative depth map value used for ranking/sorting regions. Sugimoto discloses that the image output control unit sorts the region annotation information by depth. (Sugimoto, par. [0051].) Sugimoto discloses that a depth map image is an image expressing the depth with respect to a display image in greyscale. (Sugimoto, par. [0037.]) In the grayscale image, darker regions are further from the front and brighter regions are closer to the front. The greyscale image is relative per-pixel intensity values, not depth information in the real world. Sugimoto fails to disclose that the depth information is depth information in the real world.
With respect to claim 6, Examiner cites par. [0039] of Sugimoto for disclosing that the depth information corresponds to depth information in a real world. Applicant respectfully disagrees. Sugimoto discloses that when the display image is a stereoscopic image, it is a common practice to generate the image based on binocular parallax, and in recent years, thanks to the AI technology, it has become possible to generate a depth map even based on a monocular image. (Sugimoto, par. [0039].) This merely explains how a depth map image could be generated (via binocular parallax or AI-based monocular estimation). It does not explain what kind of depth values the depth map contains. The depth information in claim 1 is depth information in a real world, i.e., relative to the real world coordinate system. The depth map in Sugimoto is a grayscale image where darker regions are further from the front and brighter regions are closer to the front, i.e., a purely relative per-pixel intensity value. Sugimoto fails to disclose that the depth information is depth information in the real world.”
In response, Examiner respectfully disagrees and submits that Sugimoto discloses a depth map as a depth image, wherein the luminous intensity of each pixel corresponds to a depth of a corresponding point in real world. This is because the depth image is generated based on binocular parallax (see [0039]).
Examiner respectfully submits that it is well known to one of ordinary skill in the art that it is well known to one of ordinary skill in the art that depth information based on binocular parallax is calculated according to formula
z
=
b
*
f
d
, whereas z is depth of field of an object in a real world scene, i.e. distance from the object to a stereoscopic camera capturing a pair of stereoscopic images, b is the distance between the left and right cameras of the stereoscopic camera, f is the focal lengths of the two cameras, and d is the binocular parallax.
As such, Applicant’s arguments with respect to Sugimoto’s teachings on page 6 are not persuasive.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 15 recites “a computer-readable medium”. However, in the state of the art, transitory signals are commonplace as a medium for transmitting computer instruction and thus, in the absence of any evidence to the contrary and give the broadest reasonable interpretation, the scope of a "computer readable medium" covers a signal per se."
A “magnetic, optical, electromagnetic, infrared, … or propagation medium” is neither a process nor a product, (i.e., a tangible “thing”) and therefore does not fall within one of the four statutory classes of § 101. Rather, a “magnetic, optical, electromagnetic, infrared, … or propagation medium” is a form of energy, in the absence of any physical structure or tangible material.
The Examiner suggests amending the claim to recite the “computer-readable medium” as “non-transitory computer-readable storage medium” to include tangible computer readable media, while at the same time excluding the intangible media such as signals, carrier waves, etc. Any amendment to the claim should be commensurate with its corresponding disclosure.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7-11, and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio et al. (US 2009/0036902 A1 – hereinafter DiMaio), Sugimoto (US 2023/0112366 A1 – hereinafter Sugimoto), and Catbas et al. (US 2022/0383478 A1 – hereinafter Catbas).
Regarding claim 1, DiMaio discloses a system for an optical imaging system (Fig. 1), the system comprising one or more processors ([0041]-[0042]; [0054] – processor 102) and one or more storage devices ([0051]; Fig. 2 – memory 240), wherein the system is configured to: obtain stereoscopic image data of a scene from a stereoscopic imaging device of the optical imaging system ([0045] – obtaining stereoscopic image data captured by a stereoscopic endoscope 140); determine depth information ([0148] – determining depth of a pointer); determine a scaling factor of an information overlay to be overlaid over the stereoscopic image data based on the depth information ([0148] – determining a scaling factor of the pointer so that its size can be determined for display according to the pointer’s depth); generate a stereoscopic composite view of the stereoscopic image data and the information overlay, with the information overlay being scaled according to the scaling factor (Fig. 22C; [0148] – generating the stereoscopic composite view of the stereoscopic image data as shown in Fig. 22C); and provide a display signal comprising the stereoscopic composite view to a stereoscopic display device (Fig. 22C; [0044]-[0045] – providing the generated stereoscopic composite view to a stereoscopic master display 104).
However, DiMaio does not disclose the system is configured to: determine depth information on at least a portion of the scene, wherein the depth information corresponds to depth information in a real world, wherein a placement depth of the information overlay is determined in the real world based on the depth information and the information overlay is at the placement depth in a world coordinate system, wherein the stereoscopic composite view is generated by forward-projecting the information overlay from the world coordinate system to image coordinates of each individual stereo image of the stereoscopic composite view based on an intrinsic calibration matrix and extrinsic calibration parameters of the stereoscopic imaging device, so that the information overlay visually appears to be drawn at a real-world depth.
Sugimoto discloses a system is configured to: determine depth information on at least a portion of a scene (Fig. 6; [0045]-[0046] – determining representative depth information for various objects of a scene); determine a scaling factor of an information overlay to be overlaid over image data based on the depth information (Fig. 8; [0062] – determining a scaling factor such as smaller size for overlay information associated with an object with larger depth information and larger size for overlay information associated with an object with smaller depth information), wherein the depth information corresponds to depth information in a real world ([0039] – the depth information obtained based on binocular parallax, thus corresponding to depth information in the real world because depth information based on binocular parallax is well known according to formula
z
=
b
*
f
d
, whereas z is depth of field of an object in a real world scene, i.e. distance from the object to a stereoscopic camera capturing a pair of stereoscopic images, b is the distance between the left and right cameras of the stereoscopic camera, f is the focal lengths of the two cameras, and d is the binocular parallax), wherein a placement depth of the information overlay is determined in the real world based on the depth information and the information overlay is at the placement depth in a world coordinate system ([0040]; [0044]-[0045] – a placement depth of the information overlay is determined in the real world as follows: (1) obtaining depths from the depth map and (2) calculating a representative depth as an average of the depths for each object, thus since the depth in (1) is in a real world, the average depth in (2) is also in the real world), wherein the stereoscopic composite view is generated by forward-projecting the information overlay from the world coordinate system to image coordinates of each individual stereo image of the stereoscopic composite view, so that the information overlay visually appears to be drawn at a real-world depth (Fig. 8; [0046]; [0062] – displaying the information overlay such as smaller size for overlay information associated with an object with larger depth information and larger size for overlay information associated with an object with smaller depth information; the information overlay for each object visually appears to be drawn at an average real-world depth of the object).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the teachings of Sugimoto into the system taught by DiMaio to enable intuitive recognition of association between a region in an image and annotation information corresponding to the region (Sugimoto: [0007]).
However, DiMaio and Sugimoto do not disclose forward-projecting from the world coordinate system to image coordinates of each individual stereo image of the stereoscopic composite view based on an intrinsic calibration matrix and extrinsic calibration parameters of the stereoscopic imaging device.
Catbas discloses forward-projecting from a world coordinate system to image coordinates based on an intrinsic calibration matrix and extrinsic calibration parameters of the stereoscopic imaging device (Fig. 3; [0055]).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the forward-projecting taught by Catbas into generating the stereoscopic composite view taught by DiMaio and Sugimoto to forward project the information overlay from the world coordinate system to image coordinates of each individual stereo image of the stereoscopic composite view more accurately.
Regarding claim 2, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above. Sugimoto also discloses the system is configured to determine the depth information based on stereoscopic image data ([0039] – generating the depth map image based on parallax of stereoscopic image).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the further teachings of Sugimoto into the system to generate the depth information when the system is not equipped with any extra depth sensor.
Regarding claim 4, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 2 above. Sugimoto also discloses the system is configured to determine per-pixel depth information of the portion of the scene based on the stereoscopic image data ([0039] – determining depth map including per-pixel depth information based on binocular parallax of stereoscopic image data); and to determine the depth information in the real world by back-projecting the per-pixel depth information, with the scaling factor being determined based on the depth information in the real world (Fig. 8; [0045]-[0046]; [0062]).
The motivation for incorporating the teachings of Sugimoto has been discussed in claim 1 above.
Regarding claim 5, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above. However, DiMaio, Sugimoto, and Catbas do not explicitly disclose the system is configured to obtain a depth sensor signal of a depth sensor of the optical imaging system, and to determine the depth information based on the depth sensor signal.
Official Notice is taken that a depth sensor to provide depth sensor signal used to determine depth information is well known in the art.
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate a depth sensor into the optical imaging system taught by DiMaio, Sugimoto, and Catbas to determine the depth information in order to reduce computing cost which would have been spent on calculating depth information from stereoscopic images.
Regarding claim 7, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above, in which Sugimoto also discloses the system is configured to overlay the information overlay over the stereoscopic image data such, that, according to the depth information, the overlay does not visually collide with elements of the portion of the scene (Fig. 8; [0062]).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the further teachings of Sugimoto above into the system so as not to block the elements in the scene from viewing.
Regarding claim 8, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 7 above, in which Sugimoto also discloses the system is configured to determine a depth profile of the portion of the scene ([0045]-[0046]; Fig. 6 – generating a depth profile as shown in Fig. 6), and to overlay the information overlay over the stereoscopic image data such, that, according to the depth profile, the overlay does not visually collide with elements of the portion of the scene (Fig. 8; [0062] – the overlay does not block the elements of the scene).
The motivation for incorporating the teachings of Sugimoto into the system has been discussed in claim 7 above.
Regarding claim 9, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 8 above, in which Sugimoto also discloses the system is configured to adapt a perceived shape of the information overlay such, that the information is adapted to the depth profile of the portion of the scene (Fig. 8 – adapting a perceived shape of overlay as shown in Fig. 8 such as sizes of the overlay is adapted to the depth profile).
The motivation for incorporating the teachings of Sugimoto into the system has been discussed in claim 1 above.
Regarding claim 10, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above, in which Sugimoto also discloses the system is configured to overlay the information overlay over the stereoscopic image data such, that, according to the depth information, the overlay visually hovers above the portion of the scene (Fig. 8).
The motivation for incorporating the teachings of Sugimoto into the system has been discussed in claim 1 above.
Regarding claim 11, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above, in which Sugimoto also discloses the system is configured to select the portion of the scene, by selecting a portion of a field of view of the stereoscopic image data that is at the center of the field of view of the stereoscopic image data, or by performing object detection or image segmentation on the stereoscopic image data and selecting the portion of the scene based on a detected object or based on a segmented portion of the stereoscopic image data, and to determine the depth information for the selected portion of the scene (Fig. 6; Fig. 8 – at least by performing object detection or image segmentation on the stereoscopic image data and selecting the portion of the scene based on a detected object or based on a segmented portion of the stereoscopic image data, and to determine the depth information for the selected portion of the scene).
The motivation for incorporating the teachings of Sugimoto into the system has been discussed in claim 1 above.
Regarding claim 13, DiMaio in view of Sugimoto and Catbas also discloses the system according to claim 1, wherein the system is configured to provide the stereoscopic display signal to at least one of a display for displaying three-dimensional content, stereoscopic ocular displays and a stereoscopic head-mounted display ([0044]; [0110]; [0131]).
Claim 14 is rejected for the same reason as discussed in claim 1 above.
Claim 15 is rejected for the same reason as discussed in claim 14 above in view of DiMaio also disclosing a computer readable medium comprising a computer program with a program code for performing the method according to claim 14 when the computer program is run on a processor ([0045]; [0051] – a program stored in memory 240 and run on a processor 102 for performing the method of claim 14).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over DiMaio, Sugimoto, and Catbas as applied to claims 1-2, 4-5, 7-11, and 13-15 above, and further in view of Jie (US 2021/0241479 A1 – hereinafter Jie).
Regarding claim 3, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 2 above. However, DiMaio, Sugimoto, and Catbas do not explicitly disclose the system is configured to determine a disparity map based on the stereoscopic image data, and to determine the depth information based on the disparity map.
Jie discloses a system is configured to determine a disparity map based on the stereoscopic image data, and to determine the depth information based on the disparity map ([0120]; Fig. 10 – determining a disparity map from a stereoscopic image data as shown in Fig. 10, and determining depth map based on the disparity map and the distance between and focal lengths of the camera lenses).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the further teachings of Jie into the system taught by DiMaio, Sugimoto, and Catbas to facilitate implementation of the calculation of the depth information, i.e. by using camera parameters such as the distance between and focal lengths of the camera lenses.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over DiMaio, Sugimoto, and Catbas as applied to claims 1-2, 4-5, 7-11, and 13-15 above, and further in view of Hong et al. (US 2015/0260972 A1 – hereinafter Hong).
Regarding claim 12, see the teachings of DiMaio, Sugimoto, and Catbas as discussed in claim 1 above. DiMaio, Sugimoto, and Catbas do not disclose the system is configured to obtain the stereoscopic image data from one or more optical imaging sensors of a microscope.
Hong discloses a system is configured to obtain stereoscopic image data from one or more optical imaging sensors of a microscope ([0008]; Fig. 1).
One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to incorporate the teachings of Hong into the system taught by DiMaio, Sugimoto, and Catbas to expand the utility of the system by processing stereoscopic images from optical imaging sensors of a surgical stereoscopic microscope.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG Q DANG whose telephone number is (571)270-1116. The examiner can normally be reached IFT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thai Q Tran can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG Q DANG/Primary Examiner, Art Unit 2484
