DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the Application No. 18/763,581 filed July 3rd, 2024.
Priority
Acknowledgment is made of applicant’s status as a divisional (DIV) of Patent Application No. 18/295,768, filed on April 4th, 2023, now U.S. Patent 12,080,012 B2, which is a DIV of Patent Application No. 17/235,159, filed on April 20th, 2021, now U.S. Patent 11,651,506 B2.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on January 20th, 2026, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS is being considered and attached by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim(s) 12, the claim recites the limitation “The method of claim 1…”. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, this limitation in this claim will be read as “The method of claim [[1]]11…”. Furthermore, claims 13-17 inherit this insufficient antecedent basis in view of their dependency to the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to “one or more hardware storage devices”. The “one or more hardware storage devices” is defined in the specification to include “Computer storage media (aka “hardware storage device”) are computer-readable hardware storage devices” and "...computer-readable physical storage media can be included in computer system components that also (or even primarily) utilize transmission media." (Originally Filed Specification Paragraphs [00148] and [00150]), which does not disavowal the claimed hardware storage devices to include transitory propagating signals per se.
The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 U.S.C 101 as covering non-statutory subject matter. The claims, as defined in the specification, cover both non-statutory subject matter and statutory subject matter. A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments by adding the limitation "non-transitory" to the claim. The examiner respectfully recommends amending the phrase “one or more hardware storage devices” to recite “one or more non-transitory computer-readable storage media” or the like to narrow the claims to cover only statutory embodiments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 8-13, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (Liu; US 2018/0302564 A1) in view Siddiqui et al. (Siddiqui; US 2016/0012567 A1).
Regarding claim 1, Liu discloses a system for facilitating low compute depth map generation to provide parallax-corrected images, the system comprising:
one or more processors (para(s). [0049], recite(s)
[0049] “…The processor 110 can process data from the images 107 and the memory 112 can store the data. In some examples, the processor 110 and the memory 112 are incorporated in to the controller 108, which is a specially programmed for efficiently and/or economically performing the logic and/or actions described herein. In some examples, the memory 112 stores instructions which when executed by the processor 110 provides execution of the logic and/or actions described herein. …”
); and
one or more hardware storage devices storing instructions that are executable by the one or more processors (“memory” in para(s). [0049]—see citation in preceding limitation above) to configure the system to:
obtain a first image of an environment (para(s). [0049] and [0051-0052], recite(s)
[0049] “FIG. 1 is a block diagram of an example environment 100 for capturing images. The environment 100 can include one or more image capturing devices 102, e.g., cameras or other device, for taking one or more images of one or more subjects 104. The image capturing device 102 can include one or more imaging sensors 1-N 106, or other detectors, for capturing images 107 of the subject 104. …”
[0051] “The image capturing device 102 can perform image registration via the different imaging sensors 1-N 106, which can view the subjects 104 from different locations, perspectives, in different modalities and/or with different fields of view (FOVs), etc. …”
[0052] “Given two images 107, the registration can find the mapping from a pixel p1 (x1, y1), in the first image I1 to a pixel p2(x2, y2) in the second image I2, e.g., finds a function f: I1.fwdarw.I2.”
, where the first of the “two images 107” is a first image of an environment (e.g., “subjects 104 from different locations”, etc.));
obtain a second image comprising(para(s). [0051-0052]—see citation in preceding limitation above—, where the second of the “two images 107” is a second image comprising information for the environment (e.g., “pixel[s]” of the environment); and para. [0079] further recites that the second image can be of “higher resolution”:
[0079] “In some examples, images of one modality have a lower resolution than the images of the other modality. For the sake of explanation, the images of lower resolution can be donated as Ilow and the images of higher resolution as Ihigh. …”
, where the “higher resolution” image “Ihigh” is at least a second image);
generate a downsampled second image, wherein the downsampled second image comprises a same image resolution as the first image (para(s). [0079] further recite(s):
[0079] “In some examples, images of one modality have a lower resolution than the images of the other modality. For the sake of explanation, the images of lower resolution can be donated as Ilow and the images of higher resolution as Ihigh. In this case, multiple pixels in Ihigh are registered with the same pixel in Ilow. There are many ways to take advantage of this. In some examples, Ihigh can be downsampled to the same resolution as Ilow before registration using any of the logic mentioned above or other image registration algorithm. The downsampled image can be denoted from Ihigh as Idown. Because the mapping from Ihigh to its downsampled, counterpart Idown is known during the downsampling process, the function denoted as d: Ihigh.fwdarw.Idown. Once Ilow and Idown are registered, resulting in the function f: Idown.fwdarw.Ilow, the logic can register Ilow to Ihigh by using the composed function f ° g. In this way, the complexity of the registration can be determined by the lower resolution of the two images.”
, where the “downsampled” higher resolution image “Ihigh”" is a downsampled second image comprising a same image resolution as the first image (i.e., the “lower resolution”));
generate a depth map of the environment by performing depth processing on the downsampled second image and the first image (para(s). [0060] and [0089], recite(s)
[0060] “The logic can register two images via a deterministic function Mz that has a variable z. When z is the depth, at different depths, the correspondence between pixels on two images are different. For example, the system can retrieve (402) depth z for p1 from image I1 (404). The system can map (406) Mz(p1, z) to p2 of Image I2 (408). The depth map itself, the result of processing data from the first and second sensors of the first modality, is an image too. In some aspects, the depth map is already registered with one of the imaging modalities.”
[0089] “FIG. 19 is a flowchart 1900 of an example logic for using downsampled images to calculate disparity/depth map and then interpolate to higher resolution. The disparity/depth map calculation can be performed in downsampled images. …”
, where the “disparity/depth map” is a depth map generated by depth processing on at least the downsampled second image (e.g., “"Image I2” recited in para. [0060] above can be image “"Idown” previously recited in para. [0079]—see previous citation in claim limitation immediately above) and the first image (e.g., “Image I1” recited in para. [0060] above can be image “Ilow” previously recited in para. [0079]—see previous citation in claim limitation immediately above)); and
generate an upsampled depth map based on the depth map and the(para(s). [0079] and [0089]—see citations in claim limitations “generate a downsampled second image…” and “generate a depth map…” above—, where “interpolate [the disparity/depth map] to a higher resolution] is generating an upsampled depth map based on the depth map (e.g., the “disparity/depth map” calculated from “downsampled images”) and the information of the second image (i.e. the “downsampled images” recited para. [0089] comprises of downsampled second image “Idown” recited in para. [0079], wherein the second image comprises of information of the environment as recited previously in para(s). [0051-0052]—see citations in “obtain a second image..” above)).
Where Liu does not specifically disclose
obtain a second image comprising texture information for the environment, the second image comprising a higher image resolution…; and
generate an upsampled depth map based on the depth map and the texture information of the second image;
Siddiqui teaches in the same field of endeavor of generating an upsampled depth map based on at least a depth map and an image from a pair of images
obtain a second image comprising texture information for the environment, the second image comprising a higher image resolution… (para(s). [0048], recite(s)
[0048] “In some aspects, the low-resolution disparity map may be upsampled to a high-resolution using linear interpolation. This method may accurately estimate the piecewise constant disparity inside objects. However, object contours in the upsampled disparity map may be blurred and aliased due to the upsampling. However, a high-resolution reference image (such as one or more of the original pair of stereo images) may be used to restore the edges of the disparity map. For example, the reference image may be high-resolution and may contain sharp edges. These sharp edges (transitions between colors of various objects in the image) may be used to sharpen the blurred edges of the upsampled disparity map. Accordingly, this method may be used to create a high-resolution disparity map which is crisp, has a high PSNR, and which is accurate for foreground objects.”
, where the “high-resolution reference image (such as one or more of the original pair of stereo images)” is a second image of a higher image resolution comprising at least texture information (e.g., “contours” and/or “edges”)); and
generate an upsampled depth map based on the depth map and the texture information of the second image (para(s). [0048]—see citation immediately above—, where the “up-sampled” “low-resolution disparity map” is an upsampled depth map based on at least the depth map (e.g., “low-resolution disparity map”) and the texture information of the second image (e.g., the “high-resolution reference image” comprising of “contours” and/or “edges” information)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Liu to incorporate obtaining a second image comprising texture information for the environment and generating an upsampled depth map based on the depth map and the texture information of the second image to restore texture information in the upsampled depth map as taught by Siddiqui above.
Regarding claim 2, Liu in view of Siddiqui discloses the system of claim 1, wherein Liu further discloses the first image and the second image are associated with a same camera modality (para(s). [0059], recite(s)
[0059] “ …In one example, the depth can be estimated from images of a pair of cameras, e.g., a pair of cameras of the same imaging modality, where the pair of cameras can work on visible light, infrared light, other modalities, or the combination thereof. For each imaging sensor in the pair of cameras, the system can establish the depth value.”
, where the “images of… a pair of cameras of the same imaging modality” is the first image and the second image (e.g., an image from each of the “pair of cameras” respectively) are associated with a same camera modality).
Regarding claim 3, Liu in view of Siddiqui discloses the system of claim 2, wherein Liu further discloses the same camera modality comprises a thermal camera modality (para(s). [0059]—see citation in claim 2 above—, where the “infrared light” modality comprises at least a thermal camera modality as recited in para(s). [0099]:
[0099] “The thermal image sensor may include cooled or uncooled infrared image detectors. …”
).
Regarding claim 8, Liu in view of Siddiqui discloses the system of claim 1, wherein Liu further discloses the depth processing comprises stereo matching (para(s). [0053] and [0091], recite(s)
[0053] “FIG. 2 is a block diagram of an example 2D-to-2D deterministic mapping of images 107 based on a parameter z. With the parameter z, the system can establish an underlining 1-to-1 mapping between the two 2D images. For the sake of explanation, each pixel in image I1 has a z value associated with it. In some examples, the parameter z is not measured in parallel with any imaging plane. Mathematically z remains the same when moving on the imaging plane or on any plane that is parallel to the imaging plane. In some examples, the parameter z may be depth, e.g., how far the source (e.g. part of an object) of a pixel is to the imaging plane. In another example, the parameter z does not have any physical meanings, but is a parameter that correctly registers two images. For example, the parameter z can be disparity value obtained from two images of the same modality but two imaging sensors.”
[0091] “…the two images of the first modality can form a stereoscopic view… ”
, where “2D-to-2D deterministic mapping of images 107” based on “disparity” from the “two images” is stereo matching).
Regarding claim 9, Liu in view of Siddiqui discloses the system of claim 1, wherein Liu further discloses the system comprises a head-mounted display (HMD) (para(s). [0119], recite(s)
[0119] “ …The display module 2900 can include an augmented reality window, augmented monitors, selective occlusion see-through head-mounted display, a retinal scanning display, and/or any other suitable display. The display module 2900 can include a picture-in-picture (PIP) display that can display images from multiple independent image sources simultaneously. In some examples, the display module 2900 can provide a transparent or partially transparent display unit, e.g., transparent OLED display, or can be a display unit with reflective surfaces, e.g., a digital projector with a partially transparent mirror/glass. The display module 2900 may be configured in a head-mounted display (HMD) configuration or a head-up display (HUD) configuration. The display module 2900 may be stereoscopic or non-stereoscopic. The display module 2900 can be optical see-through or not optical see-through.”
).
Regarding claim 10, Liu in view of Siddiqui discloses the system of claim 9, wherein Liu further discloses the first image is captured by a first camera mounted on the HMD, and wherein the second image is captured by a second camera mounted on the HMD (para(s). [0119]—see citation in claim 9 above—, where para(s). [0118] further recite(s):
[0118] “FIG. 29 is a block diagram of a cross-section view of an example image capturing device 102 for virtual reality (VR) and/or augmented reality (AR). In some examples, the image capturing device 102 can include a display module 2900, image sensors 2202, 2206 of the first imaging modality, the image sensor 2204 of the second imaging modality, lenses 2300, 2302, 2304 and computing module 2208. In some examples, the display module 2900 includes a case 2902 to provide a headset to enable virtual reality. …”
, where the “image sensors 2202, 2206” are at least a first camera and a second camera mounted on the HMD (i.e., the “display module”)).
Regarding claim 11, the claim differs from claim 1 in that the claim is the method performed by claim 1. Therefore, claim 11 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 12, the claim recites similar limitations to claim 2 and is rejected for similar rationale and reasoning (see the analysis for claim 2 above).
Regarding claim 13, the claim recites similar limitations to claim 3 and is rejected for similar rationale and reasoning (see the analysis for claim 3 above).
Regarding claim 18, the claim recites similar limitations to claim 8 and is rejected for similar rationale and reasoning (see the analysis for claim 8 above).
Regarding claim 19, the claim recites similar limitations to claim 10 and is rejected for similar rationale and reasoning (see the analysis for claim 10 above).
Regarding claim 20, the claim differs from claim 1 in that the claim is the one or more hardware storage devices in claim 1. Therefore, claim 11 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Claims 4-7 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Siddiqui as applied to claims 2 and 12 above, and further in view of Bleyer et al. (Bleyer; US 2019/0297316 A1).
Regarding claim 4, Liu in view of Siddiqui discloses the system of claim 2, wherein Liu further discloses the instructions are executable by the one or more processors to further configure the system to:
generate a parallax(para(s). [0079] and [0089]—see citations in claim 1 limitation “generate an upsampled depth map…”—, where the upsampled depth map (e.g., the interpolated “disparity/depth map”) is used to generate and “generate a depth map…” above—, where para(s). [0067], [0118], and [0132] further recite(s):
[0067] “FIG. 9 is a flowchart 900 of an example logic for when the disparity map is used with the parameter z. The logic can calibrate the imaging sensors of the first modality and the imaging sensor(s) of the second modality (902). The logic can establish the correspondence Mz between images from an imaging sensor of first modality and images of the second modality for different disparity/depth values z via computation and/or calibration (904). The logic can capture (new) images using the imaging sensors of the first modality and the imaging sensor(s) of the second modality (906). The logic can compute the disparity map for a plurality of pixels using images from the first image sensor of the first imaging modality and images of the second image sensor of the first imaging modality (908). The logic can register images from imaging sensor of first imaging modality to images from the second imaging modality based on Mz and the disparity map of parameter z (910). In some examples, the registration result can be optionally visualized/displayed (912). In some examples, the logic can optionally capture a new set of multi-modal images, etc. (914).”
[0118] “FIG. 29 is a block diagram of a cross-section view of an example image capturing device 102 for virtual reality (VR) and/or augmented reality (AR). In some examples, the image capturing device 102 can include a display module 2900, image sensors 2202, 2206 of the first imaging modality, the image sensor 2204 of the second imaging modality, lenses 2300, 2302, 2304 and computing module 2208. In some examples, the display module 2900 includes a case 2902 to provide a headset to enable virtual reality. In some examples, the case 2902 can house a mobile device 2904. In some examples, the display module 2900 can include a magnifier for right eye 2906, a magnifier for left eye 2904 and/or a head strap 2910. In some examples, the image sensors are built as an imaging module, which can be removed from the other part of the display module 2900. The mobile device 2904 can execute software to split the screen into left portion and right portion. The mobile device 2904 can display left images in the left portion of the display for the left magnifier 2908, to be viewed by the left eye of the user. The mobile device 2908 can display right images in the right portion of the display for the right magnifier 2906, to be viewed by the right eye of the user. …”
[0132] “FIG. 30 is a block diagram of a perspective view of an example image capturing device 102. In some examples, the display device 2900 can provide a headset for virtual reality (VR) and/or augmented reality (AR), as described above. The first modality and second modality can be determined based on an implementation. …”
, where “register[ing]” the images from the “image sensors” including a second image (e.g., “images from imaging sensor of first imaging modality”) is generating a parallax image (i.e., “disparity” image for display by image “registration”) by at least visualizing the second image to correspond to a user perspective (e.g., a “headset for virtual reality (VR) and/or augmented reality (AR)” comprising of a display “to be viewed by the… eye of the user”) being based on at least upsampled depth information (e.g., “disparity/depth values”) from the upsampled depth map (e.g., the “disparity map” which includes an upsampled depth as recited previously in para(s). [0079] and [0089]--see citation in claim 1 limitation “generate an upsampled depth map…” above)).
Where Liu in view of Siddiqui does not specifically disclose
generate a parallax-corrected image by reprojecting the second image to correspond to a user perspective…;
Bleyer teaches in the same field of endeavor of generating a parallax image for head-mounted displays based on at least a depth map
generate a parallax-corrected image by reprojecting the second image to correspond to a user perspective… (para(s). [0073-0074], recite(s)
[0073] “Once these digital images are obtained, then the HMD 200 performs certain transformations (also called “re-projections”) on those digital images. These transformations correct for lens distortion and other camera artifacts. Furthermore, the stereo images are re-projected onto a virtual stereo rig where both image planes lie inside a plane that is parallel to the stereo cameras' baseline. After re-projection, corresponding pixels are guaranteed to lie on the same horizontal scanline in left and right images. As a result, two “re-projected” images are formed, one for the image that was obtained by the camera 205 and one for the image that was obtained by the camera 210. Any pixels that are similar/correspond between the two re-projected images now lie on the same horizontal plain.”
[0074] “After the re-projected images are created, the HMD 200 measures any pixel disparity that exists between each of the corresponding pixels in the two images. Because the HMD 200 understands that the corresponding pixels in the two re-projected images are now in the same horizontal plain, the HMD 200 identifies that the disparity between these corresponding pixels corresponds (i.e. is proportional) with a depth measurement. Using this disparity, the HMD 200 assigns a depth value to each pixel, thus generating a depth map for any objects located in the overlapping region 235. Accordingly, the HMD 200, through the use of its multi-purposed head-tracking stereo camera pair, is able to perform both movement detection as well as depth detection.”
, where performing “re-projections” to obtain “two re-projected images” is generating at least a parallax-corrected image (e.g., one of the “two re-projected images” which “correct for lens distortion and other artifacts”) by at least reprojecting a second image to correspond to a user perspective (e.g., the head-mount display “HMD”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Liu in view of Siddiqui to incorporate generating a parallax-corrected image by reprojecting the second image to correspond to a user perspective to display/virtualize the second image in a head-mounted display for a user as taught by Bleyer above.
Regarding claim 5, Liu, as modified by Siddiqui and Bleyer, discloses the system of claim 4, wherein the instructions are executable by the one or more processors to further configure the system to:
obtain an(the examiner notes that the “” recited in this claim limitation recites similar subject matter to the “second image” recited in the claim 1 limitation “obtain a second image…” and is rejected for similar rationale and reasoning—see the rejection of claim 1 limitation “obtain a second image…” over Liu in view of Siddiqui above);
generate a reprojected(the examiner notes that the “reprojected” recited in this claim limitation recites similar subject matter to the “reprojection of the second image” recited in claim 4 above and is rejected for similar rationale and reasoning—see the rejection of claim 4 limitation “generate a parallax-corrected image…” over Liu in view of Siddiqui, and further in view of Bleyer above and below for a detailed analysis—:
wherein Liu discloses generating an image for visualization by visualizing the image (e.g., “images from imaging sensor of first imaging modality” recited in para. [0067]—see similar claim limitation in claim 4 above) to correspond to an image capture perspective (e.g., a “headset for virtual reality (VR) and/or augmented reality (AR)” comprising of a display “to be viewed by the… eye of the user” recited in paras. [0118] and [0132] of Liu—see similar claim limitation in claim 4 above) associated with the upsampled depth map (e.g., the “disparity map” which includes an upsampled depth as recited previously in para(s). [0079] and [0089]--see similar claim limitation in claim 4 above);
wherein para(s). [0048] of Siddiqui teaches the ‘visualized’ image of Liu comprises texture information for an environment as previously disclosed in claim 1 above—see the rejection of claim 1 above—, and
wherein para(s). [0073-0074] of Bleyer teaches the ‘visualized’ image of Liu comprises at least a reprojected image as disclosed previously in claim 4 above—see the rejection of claim 4 above); and
generate a parallax-corrected(the examiner notes that the “parallax-corrected” recited in this claim limitation recites similar subject matter to the “parallax-corrected image by reprojecting the second image” recited in claim 4 above and is rejected for similar rationale and reasoning—see the rejection of claim 4 limitation “generate a parallax-corrected image…” over Liu in view of Siddiqui, and further in view of Bleyer above).
Where Liu, as modified by Siddiqui and Bleyer, does not specifically disclose
obtain an additional image…, wherein the additional image is associated with a different camera modality than the first image and the second image;
Liu further teaches
obtain an additional image…, wherein the additional image is associated with a different camera modality than the first image and the second image (para(s). [0052] and [0067] recite(s) that
[0052] “…To expand the registration to register more than two images, in some examples the images can be registered pair by pair. Although the registration is described among images of two different modalities, the registration can be expanded to handle images of more than two modalities.”
[0067] “FIG. 9 is a flowchart 900 of an example logic for when the disparity map is used with the parameter z. The logic can calibrate the imaging sensors of the first modality and the imaging sensor(s) of the second modality (902). The logic can establish the correspondence Mz between images from an imaging sensor of first modality and images of the second modality for different disparity/depth values z via computation and/or calibration (904). The logic can capture (new) images using the imaging sensors of the first modality and the imaging sensor(s) of the second modality (906). The logic can compute the disparity map for a plurality of pixels using images from the first image sensor of the first imaging modality and images of the second image sensor of the first imaging modality (908). The logic can register images from imaging sensor of first imaging modality to images from the second imaging modality based on Mz and the disparity map of parameter z (910). In some examples, the registration result can be optionally visualized/displayed (912). In some examples, the logic can optionally capture a new set of multi-modal images, etc. (914).”
, where the “images from the second imaging modality” includes at least an additional image associated with a different camera modality (i.e., a “second imaging modality”) than the first image and the second image (i.e., “images from the first image sensor of the first imaging modality and images of the second image sensor of the first imaging modality”)).
Since Liu further teaches that the same image processing can be further applied to an additional image associated with a different camera modality than the first image and the second image as detailed above, it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Liu, as modified by Siddiqui and Bleyer, to substitute the second image undergoing the current process in the current claim above with an additional image associated with a different camera modality than the first image and the second image as further disclosed in Liu in order to enable 3D visualization of images captured in different modalities as taught by Liu (para(s). [0120} recite(s)
[0120] “The image capturing device 102 can capture images from the first and second modalities, coregister the images from the first modality and second modality, and enable 3D visualization of the second modality using the surrogated 3D logic described above. Similarly, 3D visualization of overlaid images of the first and the second modality can also be enabled using the surrogated 3D logic described above. The 3D effects can be created for the second modality even if only one image sensor of the second modality is used, instead of two image sensors of the second modality that are normally needed. …”
).
Regarding claim 6, Liu, as modified by Siddiqui and Bleyer, discloses the system of claim 5, wherein Liu further discloses the reprojection of the reprojected additional image for generating the parallax-corrected additional image is based on the upsampled depth information from the upsampled depth map (para(s). [0067], [0118], and [0132]—see similar claim 4 limitation “generate a parallax-corrected image…” above—, where “register[ing]” the images from the “image sensors” including an additional image (e.g., “images from the second imaging modality”) is generating the parallax-corrected image (i.e., “disparity” image for display by image “registration”) based on at least upsampled depth information (e.g., “disparity/depth values”) from the upsampled depth map (e.g., the “disparity map” which includes an upsampled depth as recited previously in para(s). [0079] and [0089]--see citation in claim 1 limitation “generate an upsampled depth map…” above)).
Regarding claim 7, Liu, as modified by Siddiqui and Bleyer, discloses the system of claim 5, wherein Bleyer further teaches the different camera modality comprises a low light camera modality (para(s). [0151], recite(s)
[0151] “As discussed earlier, the HMD includes at least two cameras (though more than two may be used), both of which are configured to obtain one or more visible light images of the low light environment as well as one or more IR light images of the low light environment. Notably, these visible light and IR light images may be used to track movements of the HMD. Of course, a single image may capture both visible light and IR light.”
, where the “visible light images of the low light environment” are image captured by a low light camera modality).
Regarding claim 14, the claim recites similar limitations to claim 4 and is rejected for similar rationale and reasoning (see the analysis for claim 4 above).
Regarding claim 15, the claim recites similar limitations to claim 5 and is rejected for similar rationale and reasoning (see the analysis for claim 5 above).
Regarding claim 16, the claim recites similar limitations to claim 6 and is rejected for similar rationale and reasoning (see the analysis for claim 6 above).
Regarding claim 17, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/MING Y HON/Primary Examiner, Art Unit 2666
Prosecution Insights