DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks pages 8-10, filed 07/23/2026, with respect to the rejection of amended claim(s) 1, 11, and 12 under 35 U.S.C. 103, specifically in regards to prior art reference Nguyen, have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Applicant’s arguments, see Remarks page 10, filed 07/23/2026, with respect to the rejection of amended claim(s) 1, 11, and 12 under 35 U.S.C. 103, specifically in regards to prior art reference Chen, have been fully considered but they are not persuasive.
On page 10 of Remarks, Applicant argues:
PNG
media_image1.png
240
823
media_image1.png
Greyscale
Examiner respectfully disagrees.
Paragraphs 0019-0020 of Chen disclose: “An exemplary embodiment of the present disclosure provides an automatic focusing method adapted to an image capturing device with multiple cameras. The automatic focusing method captures a scene through the cameras to obtain multiple images corresponding to the cameras. The scene can has single one object or multiple objects, and the object can be a background, animal, landmark, human, tree, cloud, mountain, or waterfall. Next, the automatic focusing method can generate multiple depth maps according to the images, wherein each depth map is generated by the arbitrary two images… Resolutions of the single one object or the multiple objects in the depth maps are different from each other. According to the resolutions of the single one object or the multiple objects in the depth maps, the automatic focusing method selects depth information of the single one object or the multiple objects in the depth maps to generate a merged depth map.”, wherein Chen discloses a method for generating a merged image from a plurality of captured depth images based on the resolutions of the objects present in the depth images.
Paragraph 0021 of Chen further discloses “Concretely, in one exemplary embodiment of the present disclosure, for the single one object or each of the multiple objects, if the object appears in portion or all of the depth maps, the automatic focusing method selects the depth information of the object in the depth map which has a maximum resolution of the object as depth information of the object in the merged depth map. For the single one object or each of the multiple objects, if the object merely appears in one of the depth maps, the automatic focusing method selects the depth information of the object in the depth map as the depth information of the object in the merged depth map.”, wherein the generating of the merged image is performed by comparing the resolutions of an object region present in the captured depth maps, and selecting the depth values for the object region from the depth map with the highest resolution for the object. The identifying/selection of depth values based on a captured depth map with a highest maximum resolution constitutes the identifying/selection of depth values based on a candidate depth image with the highest reliability score.
Thus, Tanaka in view of Bellows and Chen, as is further disclosed below, discloses the claim 1 limitations: “select, for each pixel and based on the method-specific reliability scores S(i) for the candidate distance values expressed by the plurality of candidate depth images by comparing, for that pixel, the reliability scores S(i) of the candidate distance values across the plurality of candidate depth images, and identifying a distance value for that pixel from a candidate depth image having the highest reliability score S(i) from among the plurality of candidate depth images;”.
As per claim(s) 11 & 12, arguments made in rejecting claim(s) 1 are analogous.
Claim Objections
Claims 1 and 11-12 are objected to because of the following informalities:
Regarding claim 1, the claim limitation “identify the respective elements x representing characteristics of the object or the measurement depth image;”, should be corrected to “identify
Regarding claim 11, the claim limitation “identifying the respective elements x representing characteristics of the object or the measurement depth image;”, should be corrected to “identifying
Regarding claim 12, the claim limitation “identifying the respective elements x representing characteristics of the object or the measurement depth image;”, should be corrected to “identifying .
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka (US2017148137A1) in view of Bellows et al. (US10965929B1) hereinafter referenced as Bellows, and Chen et al. (US2015156399A1) hereinafter referenced as Chen.
Regarding claim 1, Tanaka discloses: A distance information generation apparatus comprising: circuitry configured to: obtain, from an image capturing apparatus, data regarding a measurement depth image that expresses distance values to an object (Tanaka: Abstract);
generate a plurality of candidate depth images by performing one or more of a plurality of upsampling methods on the measurement depth image, wherein performing an upsampling method of the plurality of upsampling methods comprises using the distance values expressed by the measurement depth image, wherein each candidate depth image of the plurality of candidate depth images expresses candidate distance values (Tanaka: Figure 2; 0050: “Then in step S21, the up-sampling unit 133 performs the first up-sampling processing on the depth image data 302 . In this embodiment, the nearest-neighbor interpolation is used for the first up-sampling processing.”; 0054: “Then in step S22, the up-sampling unit 133 performs the second up-sampling on the depth image data 302 . In this embodiment, the bilinear interpolation is used for the second up-sampling processing. Any method can be used for the algorithm of the second up-sampling, but the algorithm must be different from the algorithm of the first up-sampling.”);
determine, for each pixel or for each region of the measurement depth image, a reliability for the candidate distance values expressed by the plurality of candidate depth images (Tanaka: 0058: “Then in step S 23 , the confidence data determination unit 134 determines the level of confidence in each pixel of the depth image data, whereby the confidence data is generated. As described above, in this embodiment, the confidence data is binary (reliable or unreliable). The confidence data determination unit 134 compares the depth values of the same pixel portions of the 2 depth image data, generated after the first up-sampling and after the second up-sampling, determines that the depth value is unreliable if the difference is greater than a threshold, and determines that the depth value is reliable if the difference is the threshold or less.”);
select, for each pixel, an identifying distance value for that pixel, based on the determined reliability for the candidate distance values (Tanaka: 0064: “In step S 24 , the depth map correction unit 135 corrects the depth image data after the up-sampling, based at least partially on the confidence map calculated in step S 23.” ); and
generate an output depth image having pixels assigned the selected distance values so that the output depth image has higher accuracy than the measurement depth image or the plurality of candidate depth images (Tanaka: 0069-0071: “According to this embodiment, the up-sampling is performed using different up-sampling methods, and the difference between the pixel values after each up-sampling is analyzed, whereby the depth boundary (region where level of confidence in the pixel values is low) can be accurately extracted…the level of confidence that is required to correct an error in the depth boundary portion caused by up-sampling can be correctly determined, therefore the depth image data can be corrected at higher precision.”).
Tanaka does not disclose expressly: identify the respective elements x representing characteristics of the object or the measurement depth image;
determine, for each pixel or for each region of the measurement depth image, a reliability for the candidate distance values expressed by the plurality of candidate depth images, wherein the reliability is computed, for each upsampling method i, as a method-specific reliability score S(i) that is equal to a sum of individual reliability scores s-i(x), wherein each individual reliability score s-i(x) represents a suitability of the i-th upsampling method for a corresponding identified element x when the i-th upsampling method is applied.
Bellows discloses: A distance information generation apparatus comprising: circuitry (Bellows: Abstract) configured to: identify respective elements x representing characteristics of an object or a measurement depth image (Bellows: Col 9: Lines: 47-57: “The video processing device 200 also applies texture detection filters, to provide a confidence value at every pixel for the reliability of the depth map (blocks 314 and 318). Texture filters may detect conditions such as: very flat or homogenous texture; highly speckled textures; low signal-to-noise ratio (i.e. due to poorly lit scenes, or high sensor noise); and so forth… Features such as these would be assigned lower confidence values.”;
Col 9-10: Lines 58-8: “The video processing device 200 also generates a second estimate for the depth of the live scene (e.g., a second depth map) based on measurements (e.g., ToF measurements) performed by the depth sensor 118 (block 326 ). Texture filters are also applied to the ToF depth map for a confidence rating (block 328 )…These filters can include both temporal and spatial filters to detect noise, looking particularly for large depth discontinuities across small regions. Textures may also be detected in the RGB domain to vote down the confidence in the IR depth measurement; for example, depth discontinuities detected on a surface that appears flat in the RGB domain are likely noise. These types of discontinuities are common to ToF sensors today and provide disorienting errors in the MR scene to the user. By fusing the information readily available in the two domains, the depth estimate can be highly refined.”; Wherein the texture detection in the RGB and depth map images identify object characteristics.);
determine, for each pixel or for each region of a confidence-based depth map, a reliability for the candidate distance values expressed by the plurality of candidate depth images, wherein the reliability is computed, for each depth mapping method i, as a method-specific reliability score S(i) that is equal to a sum of individual reliability scores s-i(x), wherein each individual reliability score s-i(x) represents a suitability of the i-th depth mapping method for a corresponding identified element x when the i-th depth mapping method is applied (Bellows: Col 1: Lines 48-60: “In embodiments, the video processing device may be configured to: generate a first depth map based on time-of-flight measurements detected by a depth sensor of a head mounted device; generate a second depth map based on disparity mapping from stereo imagery detected by the stereoscopic camera system of the head mounted device; determine confidence values for respective pixel locations in the first depth map and the second depth map based on texture recognition; blend the first depth map and the second depth map into a combined depth map based on the confidence values of the respective pixel locations in the first depth map and the second depth map;”;
Col 9-10: Lines 47-8: “The video processing device 200 also applies texture detection filters, to provide a confidence value at every pixel for the reliability of the depth map (blocks 314 and 318). Texture filters may detect conditions such as: very flat or homogenous texture; highly speckled textures; low signal-to-noise ratio (i.e. due to poorly lit scenes, or high sensor noise); and so forth… Features such as these would be assigned lower confidence values…Texture filters are also applied to the ToF depth map for a confidence rating (block 328 )…These filters can include both temporal and spatial filters to detect noise, looking particularly for large depth discontinuities across small regions. Textures may also be detected in the RGB domain to vote down the confidence in the IR depth measurement…By fusing the information readily available in the two domains, the depth estimate can be highly refined.”; Wherein the texture elements/features are detected and aggregated, or summed, in order to determine a confidence value for each pixel location within each candidate depth map.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the reliability score calculation disclosed by Tanaka with the methods for calculating confidence values for each deep map method based on texture detection taught by Bellows by calculating confidence values based on color images and the candidate up-sampling images. The suggestion/motivation for doing so would have been “The video processing device 200 /controller 202 can be configured to determine confidence values for respective pixel locations in the first depth map and the second depth map based on texture recognition/detection. In some embodiments, the confidence values are based on recognition of textures that are unlikely to produce a good estimate in the different sensor domains.” (Bellows: Col 8: Lines 29-32; Wherein the reliability of depth map generation methods vary depending on imaging conditions.). Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results.
Tanaka in view of Bellows does not disclose expressly: select, for each pixel and based on the method-specific reliability scores S(i) for the candidate distance values expressed by the plurality of candidate depth images by comparing, for that pixel, the reliability scores S(i) of the candidate distance values across the plurality of candidate depth images, and identifying a distance value for that pixel from a candidate depth image having the highest reliability score S(i) from among the plurality of candidate depth images.
Thus, Tanaka in view of Bellows does not disclose expressly: the selection of distance values, for each pixel in the output depth image, by selecting the corresponding distance value from the candidate depth images having the highest reliability score.
Chen discloses: a method of generating a merged depth map, constructed based on the merging of multiple depth maps (Chen: Abstract). Wherein the selection of distance values, for each object in the output depth image, is done by selecting the corresponding distance values from the candidate depth images having the highest resolution (Chen: 0019: “The automatic focusing method captures a scene through the cameras to obtain multiple images corresponding to the cameras…the automatic focusing method can generate multiple depth maps according to the images, wherein each depth map is generated by the arbitrary two images. The depth map has depth information of the single one object or at least one of the multiple objects (if the two images which form the depth map have the single one object or at least the same one of multiple objects), or does not have the depth information of the single one object or all of the multiple objects (if the two images which form the depth map do not have the single one object or at least the same one of multiple objects).”
0021: “Concretely, in one exemplary embodiment of the present disclosure, for the single one object or each of the multiple objects, if the object appears in portion or all of the depth maps, the automatic focusing method selects the depth information of the object in the depth map which has a maximum resolution of the object as depth information of the object in the merged depth map.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of merging depth maps based on the maximum resolution taught by Chen by selecting the candidate image depth values disclosed by Tanaka in view of Bellows with the maximum reliability/confidence scores. The suggestion/motivation for doing so would have been “For the single one object or each of the multiple objects, if the object appears in the portion or all of the depth maps, the automatic focusing method compares the resolutions of the object in the depth maps, and selects the depth information of the object in the depth map which has a maximum resolution of the object as the depth information of the object in the merged depth map. For the single one object or each of the multiple objects, if the object merely appears in one of the depth maps, the automatic focusing method selects the depth information of the object in the depth map as the depth information of the object in the merged depth map” (Chen: 0023; Wherein the presence of multiple candidate values allows for the optimization of depth values). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows with Chen to obtain the invention as specified in claim 1.
As per claim(s) 11, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 12, arguments made in rejecting claim(s) 1 are analogous. In addition, paragraph 0125 of Tanaka discloses a non-transitory, computer readable storage medium containing a computer program which is executed by a computer.
Claim(s) 3 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows and Chen, and further in view of Xie et al. (Holistically-Nested Edge Detection) hereinafter referenced as Xie.
Regarding claim 3, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to determine the reliability on a basis of an analysis result according to a Convolutional Neural Network for a color image captured at a corresponding field of view.
Xie discloses: circuitry configured to extract edges on a basis of an analysis result according to a Convolutional Neural Network for a color image captured at a corresponding field of view (Xie: Section 5. Conclusion: “In this paper, we have developed a new convolutional-neural-network-based edge detection system that demonstrates state-of-the-art performance on natural images at a speed of practical relevance (e.g., 0.4 seconds using GPU and 12 seconds using CPU).”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the edge Detection CNN taught by Xie by extracting the edges from the color and candidate depth images disclosed by Tanaka in view of Bellows and Chen prior to performing reliability analysis. The suggestion/motivation for doing so would have been “automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of .782) and the NYU Depth dataset (ODS F-score of .746), and do so with an improved speed (0.4s per image) ...” (Xie: Abstract). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Xie to obtain the invention as specified in claim 3.
Regarding claim 5, Tanaka in view of Bellows, Chen, and Xie discloses: The distance information generation apparatus according to claim 3, wherein the circuitry is configured to perform object recognition on a basis of the color image and increases the reliability of a distance value according to an upsampling method registered as being suitable for an estimated shape (Xie: Abstract: “Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to resolve the challenging ambiguity in edge and object boundary detection.”)
(Bellows: Col 9-10: Lines 47-8: “The video processing device 200 also applies texture detection filters, to provide a confidence value at every pixel for the reliability of the depth map (blocks 314 and 318). Texture filters may detect conditions such as: very flat or homogenous texture; highly speckled textures; low signal-to-noise ratio (i.e. due to poorly lit scenes, or high sensor noise); and so forth… Features such as these would be assigned lower confidence values…Texture filters are also applied to the ToF depth map for a confidence rating (block 328 )…These filters can include both temporal and spatial filters to detect noise, looking particularly for large depth discontinuities across small regions. Textures may also be detected in the RGB domain to vote down the confidence in the IR depth measurement…By fusing the information readily available in the two domains, the depth estimate can be highly refined.”.).
Claim(s) 4 are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows, Chen, and Xie, and further in view of Bamji et al. (US2011285910A1) hereinafter referenced as Bamji.
Regarding claim 4, Tanaka in view of Bellows, Chen, and Xie discloses: The distance information generation apparatus according to claim 3, wherein the circuitry is configured to extract an edge region for an object on a basis of the color image (Xie: Abstract).
Tanaka in view of Bellows, Chen, and Xie does not disclose expressly: wherein the circuitry is configured to, by whether in the edge region or not, causes the reliability to change.
Bamji discloses: circuitry is configured to extract an edge region for an object on a basis of the color image and, by whether in the edge region or not, causes the reliability to change. (Bamji: 0063-0065: “At step 460 , an edge map is created by labeling each RGB-pixel as being on an edge or not being on an edge. A pixel is declared to be an edge based upon the local smoothness of its depth surface. In one embodiment, the variance of the depth values in a set-size neighborhood of the pixel is computed. If the variance is above a desired threshold, the RGB pixel is labeled as an edge pixel. The collection of such label declarations is the edge map…According to the edge map, depth pixels that are considered edges are highly refined, preferably using a modified bilateral filter. Bilateral filtering tends to require substantial computation time and will use input from all three sources, RGB, Z, and confidence. Those pixels not considered edges are also refined, preferably with a low-computation time method, such as a median filter or a box filter, using only the Z information. The result is the up-sampled depth estimate.”; Wherein the reliability of the filters is determined based on the edge regions.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the known technique of selecting a depth image refinement filter based edge regions as taught by Bamji in order to modify the candidate depth image reliability calculations disclosed by Tanaka in view of Bellows, Chen, and Xie based on extracted edges. The suggestion/motivation for doing so would have been “The refined estimate provides a value that can resolve ambiguities caused by imprecise depth measurements (relative to color measurements) with a general notion of clustering pixels which are already near each other in the joint-space of color and spatial location (X, Y, and Z). It will be the case that for some adjacent edge pixels PRGB i and PRGB i , the depth estimates Zi and Zj will contain similar values where a true depth discontinuity is present. Applicants' refinement method will provide more accurate depth estimates by use of RGB (color) information, thus inserting a true discontinuity where one was not indicated.” (Bamji: 0066). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows, Chen, and Xie with Bamji to obtain the invention as specified in claim 4.
Claim(s) 6 & 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows and Chen, and further in view of Jung et al. (US 2015/0015569 A1), hereinafter referenced as Jung.
Regarding claim 6, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to, on a basis of at least one of a color or a surface reflection characteristic for an object, evaluate the reliability of a distance value in the measurement depth image and incorporate the reliability of the distance value in a reliability calculation.
Jung discloses: A method for upsampling a low-resolution depth image based on a high-resolution color image corresponding to the low-resolution depth image (Jung: Abstract). Wherein the circuitry is configured to, on a basis of at least one of a color or a surface reflection characteristic for an object, evaluate the reliability of a distance value in the measurement depth image and incorporate the reliability of the distance value in a reliability calculation (Jung: Figure 13; 0062-0064: “FIG. 13 is a flowchart for explaining a method of processing a depth image according to an embodiment of the present inventive concept. Referring to FIG. 13, in operation 1300 , a high-resolution color image and a low-resolution depth image are input…In operation 1302 , a feature vector may be generated based on a depth distribution of the low-resolution depth image. A hole pixel to be subject to upsampling or filtering is determined from the low-resolution depth image. A feature vector may be generated based on a distribution characteristic indicating weights of depth values of the neighboring pixels with respect to the hole pixel. The weight may be determined according to a distance between the hole pixel and the neighboring pixels or a color difference value between the hole pixel and the neighboring pixels, or both. In other words, it is determined that the weight increases as the distance decreases and the color difference value decreases…In operation 1304 , a filter to upsample the low-resolution depth image may be selected by classifying the feature vector according to a previously learnt classifier.”; Wherein the reliability of an upsampling method’s distance value is determined based on a classifier.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the classifier taught by Jung for the determination of candidate distance value reliability score disclosed by Tanaka in view of Bellows and Chen. The suggestion/motivation for doing so would have been “As described with reference to FIGS. 2 to 5, a type of filter that will be effective for upsampling may be different according to a difference in a distribution of a depth image…an upsampling filter according to a distribution characteristic of a depth image is learnt so that an effective upsampling filter may be selected according to a difference in the distribution characteristic of a depth image” (Jung: 0037). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Jung to obtain the invention as specified in claim 6.
Regarding claim 13, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the element (x) associated with the measurement depth image is at least one of: an edge region, a shape, a color, reflection characteristic, or a relative speed of an object.
Jung discloses: A method for upsampling a low-resolution depth image based on a high-resolution color image corresponding to the low-resolution depth image (Jung: Abstract). Wherein an element (x) associated with a measurement depth image is at least one of: an edge region, a shape, a color, reflection characteristic, or a relative speed of an object (Jung: Figure 13; 0062-0064: “FIG. 13 is a flowchart for explaining a method of processing a depth image according to an embodiment of the present inventive concept. Referring to FIG. 13, in operation 1300 , a high-resolution color image and a low-resolution depth image are input…In operation 1302 , a feature vector may be generated based on a depth distribution of the low-resolution depth image. A hole pixel to be subject to upsampling or filtering is determined from the low-resolution depth image. A feature vector may be generated based on a distribution characteristic indicating weights of depth values of the neighboring pixels with respect to the hole pixel. The weight may be determined according to a distance between the hole pixel and the neighboring pixels or a color difference value between the hole pixel and the neighboring pixels, or both. In other words, it is determined that the weight increases as the distance decreases and the color difference value decreases…In operation 1304 , a filter to upsample the low-resolution depth image may be selected by classifying the feature vector according to a previously learnt classifier.”; Wherein the reliability of an upsampling method’s distance value is determined based on neighboring pixel color differences, which are associated with characteristics of an input depth image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the classifier taught by Jung for the determination of candidate distance value reliability score disclosed by Tanaka in view of Bellows and Chen. The suggestion/motivation for doing so would have been “As described with reference to FIGS. 2 to 5, a type of filter that will be effective for upsampling may be different according to a difference in a distribution of a depth image…an upsampling filter according to a distribution characteristic of a depth image is learnt so that an effective upsampling filter may be selected according to a difference in the distribution characteristic of a depth image” (Jung: 0037). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Jung to obtain the invention as specified in claim 13.
Claim(s) 7 & 10 are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows and Chen, and further in view of Mirbach et al. (US 2015235351 A1), hereinafter referenced as Mirbach.
Regarding claim 7, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to identify a pixel for which a distance value in the measurement depth image is obtained but the distance value is not within a predetermined range or pixels for which a predetermined number is not reached, and adjusts the reliability for such pixels.
Mirbach discloses: identifying a pixel for which a distance value in the measurement depth image is obtained but the distance value is not within a predetermined range or pixels for which a predetermined number is not reached, and adjusting the reliability for such pixels (Mirbach: 0018-0020: “If a valid depth value is not available for a given pixel, the corresponding pixel in the enhanced depth image will contain a depth value obtained exclusively by application of the first filter, i.e. the cross bilateral filter. Preferably, the first filter is configured to exclude contributions of pixels containing no depth value or an invalid depth value. The second filter may also be configured to exclude contributions of pixels containing no depth value or an invalid depth value.”;
0049: “In order to cope with regions of invalid pixels depth image, a so-called "occlusion map" V is introduced. V is a mask taking the value of zero for all pixel having no valid depth value and 1 for all other pixels.”; Wherein pixels with invalid depth data are labeled as invalid by the occlusion map, and thus labeled as unreliable for the calculation of upsampled values.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique taught by Mirbach of labeling the invalid pixels into Tanaka in view of Bellows and Chen by labeling pixels with invalid depth values present in the measurement depth image, prior to performing the up-sampling methods. The suggestion/motivation for doing so would have been “the first filter is configured to exclude contributions of pixels containing no depth value or an invalid depth value. The second filter may also be configured to exclude contributions of pixels containing no depth value or an invalid depth value.” (Mirbach: 0019-0020; Wherein the exclusion of invalid pixels assists in the increasing of accuracy as the propagation of invalid/unreliable depth values is reduced.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Mirbach to obtain the invention as specified in claim 7.
Regarding claim 10, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to associate invalid data for, among the output depth image, a pixel for which a reliability does not satisfy a predetermined criterion.
Mirbach discloses associating invalid data for a pixel for which a reliability does not satisfy a predetermined criterion (Mirbach: Abstract: “pixels in the depth image containing no depth value or an invalid depth value”; 0049: “In order to cope with regions of invalid pixels depth image, a so-called "occlusion map" V is introduced. V is a mask taking the value of zero for all pixel having no valid depth value and 1 for all other pixels.”; Wherein pixels with invalid depth data are labeled as invalid by the occlusion map).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique taught by Mirbach of labeling the invalid pixels into Tanaka in view of Bellows and Chen by labeling pixels with invalid depth values present in the up-sampled depth image. The suggestion/motivation for doing so would have been “the first filter is configured to exclude contributions of pixels containing no depth value or an invalid depth value. The second filter may also be configured to exclude contributions of pixels containing no depth value or an invalid depth value.” (Mirbach: 0019-0020; Wherein the exclusion of invalid pixels assists the increasing of accuracy of methods using the images by not including invalid data). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Mirbach to obtain the invention as specified in claim 10.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows and Chen, and further in view of Ilic et al. (US 20160328827 A1), hereinafter referenced as Ilic.
Regarding claim 8, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to: obtain a measurement value from a motion sensor incorporated in at least one of: the object or an image capturing apparatus used to obtain the measurement depth image, and suspend processing for determining the reliability in a time period in which a magnitude of motion obtained from the measurement value exceeds a threshold.
Ilic discloses: a method for obtaining a measurement value from a motion sensor incorporated in an image capturing apparatus, and suspends processing in a time period in which a magnitude of motion obtained from the measurement value exceeds a threshold (Ilic: 0348: “A stop condition may be implemented by detecting an output of a motion sensor on the portable electronic device as the device moves and/or tilts. In some embodiments, the stop condition may occur when the portable electronic device is no longer positioned to capture the current scene, such as when the device tilts by more than a threshold angular amount from the orientation used to capture image frames or moves at a speed that exceeds a rate at which the camera can capture image frames with motion blur exceeding a threshold.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the motion sensor present in the portable electronic device taught by Ilic into the imaging optical system disclosed by Tanaka in view of Bellows and Chen. The suggestion/motivation for doing so would have been “the stop condition may occur when the portable electronic device is no longer positioned to capture the current scene…or moves at a speed that exceeds a rate at which the camera can capture image frames with motion blur exceeding a threshold.” (Ilic: 0348). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Ilic to obtain the invention as specified in claim 8.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Bellows and Chen, and further in view of Girdzijauskas et al. (US2014205023A1) hereinafter referenced as Girdzijauskas.
Regarding claim 9, Tanaka in view of Bellows and Chen discloses: The distance information generation apparatus according to claim 1.
Tanaka in view of Bellows and Chen does not disclose expressly: wherein the circuitry is configured to perform smoothing processing at, in the output depth image, a boundary between regions in which results upsampled by different methods are employed.
Girdzijauskas discloses: the process of additionally smoothing an upsampled depth image (Girdzijauskas: 0063-0064: “additional smoothing of the updated and upsampled auxiliary information map can be done in order to suppress and combat bluring artifacts. Such a smoothing of the updated pixel values in the upsampled auxiliary information map can be performed by pixel value filtering using, for instance, bilateral filtering…the auxiliary information map is a depth map comprising multiple pixels having a respective depth value”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the pixel value filtering technique taught by Girdzijauskas on the up-sampled depth image disclosed by Tanaka in view of Bellows and Chen. The suggestion/motivation for doing so would have been “additional smoothing of the updated and upsampled auxiliary information map can be done in order to suppress and combat bluring artifacts” (Girdzijauskas: 0063). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Bellows and Chen with Girdzijauskas to obtain the invention as specified in claim 9.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights