DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 10-13, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sagong et al. (U.S. Patent Application Publication No. 2024/0290007), hereinafter referenced as Sagong, in view of (U.S. Patent Application Publication No. 2022/0353530), hereinafter referenced as McCready.
Regarding claim 1, Sagong teaches A method comprising: generating a plurality of feature maps based on a plurality of images triggered to capture at a same time, the plurality of images having a plurality of view perspectives (paragraph 61 teaches “device may obtain a plurality of images (e.g., a pair of images). For example, the electronic device may obtain a stereo image. The electronic device may capture a stereo image through a stereo camera (e.g., a camera 1010 of FIG. 10) of the electronic device” and paragraph 62 teaches “may estimate NSR data for representing a 3D scene based on input images. For example, the electronic device may extract a feature map (e.g., a 2D color feature map) from a pair of input images (e.g., a first image 201-1 and a second image 201-2) through the stereo-specific encoder 210. The electronic device may extract pyramid level color feature maps”); this shows estimating/generating feature maps based on two/stereoscopic/plurality of images which have a plurality of view perspective and are captured at the same time since they are stereoscopic; generating a depth map based on the plurality of feature maps (paragraph 65 teaches “electronic device of one or more embodiments may apply stereo rectification to obtain parallel input images without distortion. In this case, the input images may correspond to viewpoints that have parallel epipolar lines and are also positioned at the same height in a 3D space. As stereo matching-based depth estimation, a pair of input images may be images captured from viewpoints adjacent to each other, and thus the cost volume 203 may be computed for the 2D feature maps described above. The cost volume 203, which is information corresponding to the depth of a scene”); cost volume computed for feature maps which corresponds to depth and stereo matching-based depth estimation here show depth map generation process which is based on feature maps due to cost volume computed for feature maps; and generating an image based on the depth map and the plurality of images, the image having a view perspective not included in the plurality of view perspectives (paragraph 70 teaches “Using the models, the electronic device of one or more embodiments may reconstruct a scene image of a new viewpoint in a 3D space using a relatively small number of images (e.g., at least one pair of images)”); this shows new viewpoint (having a view perspective not included in the plurality of view perspectives[even though based on images such e.g. the pair]) image being generated and is based on the depth map because comes in a step after.
However, Sagong fails to explicitly teach generating a layered depth map; generating an image based on the layered depth map.
However, McCready explicitly teaches generating a layered depth map (McCready, paragraph 67 teaches “a “multilayer scene representation” (MSR) is a plurality of scene layers describing a scene volume. An MSR may optionally describe the camera perspective used by each or all scene layers. An MSR includes at least two scene layers, each of the scene layers including a depth map, a color map, and a transparency map. However, in general, an MSR may include any number greater than two of scene layers, such as 2-4 scene layers, 3-5 scene layers, 5-10 scene layers, 3-10 scene layers, or more.”); multiple scene layers with each including depth map shows layered depth map generation; generating an image based on the layered depth map (McCready, paragraph 67 teaches “advantageous feature of 6 DoF video playback is that the viewer may move their viewing perspective inside of a range and when doing so they should be provided with a plausible view of the reconstructed 3D scene from that new perspective” and paragraph 92 teaches “a packed MSR video 610 for creating rendered perspectives ready to be passed to a 6 DoF display”); this shows new perspective image would be generated and is based on the layered depth map because MSR video 610 for 6 DoF (which enables new perspective) happens after the generation of layered depth map. McCready is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of having layered depth map and new viewing perspectives synthesized. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Sagong's invention with the layering and generating techniques of McCready to strike an optimum balance between good scene description, representation, and encoding within the scene layers (McCready, paragraph 67). This would lead to a more efficient system overall due to the balance.
Regarding claim 2, the combination of Sagong and McCready teaches wherein generating the plurality of feature maps includes encoding and downsampling the plurality of images (Sagong, paragraph 86 teaches “may generate a bottom-up feature map corresponding to each pyramid level by sequentially performing downsampling in the feature pyramid network”, paragraph 89 teaches “In a stereo-specific encoder 500, a 2D neural network 510 may include one or more convolution layers. The electronic device may extract a feature map” and paragraph 120 teaches “The processor 1020 may generate images of the same or similar format by preprocessing images captured by different cameras to input the images to a stereo-specific encoder described above… apply downsampling to an image having a large size to match it to adjust the size of the image”); generating feature maps by performing downsampling as well as stereo-specific encoder used to generate the feature map shows generation of feature maps includes encoding and downsampling which is done to the plurality of images that are input.
Regarding claim 3, the combination of Sagong and McCready teaches wherein the plurality of feature maps is a feature pyramid, (Sagong, paragraph 86 teaches “extract a color feature map for each pyramid level based on the feature pyramid network”); each level outputting feature map shows plurality of feature maps is a feature pyramid; and the feature pyramid is a structured arrangement of feature maps having multiple scales (paragraph 86 teaches “FIG. 4 shows color feature maps 402-1 of three levels, including a zeroth (Oth) pyramid level feature map 421, a first pyramid level feature map 422, and a second pyramid level feature map 423, for the first image 201-1. Similarly, the electronic device may also generate color feature maps 402-2 for the second image 201-2”); this alongside the references 419-1 and 402-2 shows feature pyramid with different levels (in structured arrangement) of feature maps having multiple scales/sizes.
Regarding claim 10, the combination of Sagong and McCready teaches wherein generating the image includes projecting the plurality of images onto layers of the layered depth map (McCready, paragraph 107 teaches “neutral camera perspective 1108 is projection of the camera perspective into a 3D rendering context.”, paragraph 75 teaches “As the total number of scene layers is fixed, this strategy helps ensure that any visual features in the scene that are likely to be visible from and around the target camera perspective are represented in a scene layer before the MSR's capacity for representation is filled up” and paragraph 76 teaches “Surfaces may often contain areas wherein a transition between scene layers is required. In such regions, it is advantageous to have at least one vertex of overlap (“seam”) between the projection of the scene layers. In practice, it is preferred to have two vertices of overlap between projected scene layers”); this shows generation of image including projecting images onto layers by showing camera perspective projected into 3D rendering context as well as camera perspective represented in scene layer and since aforementioned the layers each including depth map (making the layered depth map) shows these would be layers of the layered depth map. The same motivations used in claim 1 apply here in claim 10.
Regarding claim 11, the apparatus claim 11 recites similar limitations as method claim 1, and thus is rejected under similar rationale. In addition, Sagong fig. 10 shows apparatus 1000 with processor 1020 and memory 1030 and paragraph 128 teaches “instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler”.
Regarding claim 12, the apparatus claim 12 recites similar limitations as method claim 2, and thus is rejected under similar rationale.
Regarding claim 13, the apparatus claim 13 recites similar limitations as method claim 3, and thus is rejected under similar rationale.
Regarding claim 19, the apparatus claim 19 recites similar limitations as method claim 10, and thus is rejected under similar rationale.
Regarding claim 20, the non-transitory computer-readable storage medium claim 20 recites similar limitations as product/processor claim 1, and thus is rejected under similar rationale. In addition, Sagong paragraph 129 teaches “to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media” and fig. 10 shows the memory 1030 and processor 1020 for performing such.
Claim(s) 4-5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Sagong and McCready as applied to claims 1 and 11 above, and further in view of Guizilini et al. (U.S. Patent Application Publication No. 2022/0148204), hereinafter referenced as Guizilini.
Regarding claim 4, the combination of Sagong and McCready teaches wherein generating the layered depth map includes iteratively decoding the plurality of feature maps, and iteratively generating an intermediate layered depth map based on the decoded plurality of feature maps
However Guizilini teaches wherein generating the layered depth map includes iteratively decoding the plurality of feature maps, (Guizilini, paragraph 39 teaches “Accordingly, while the depth decoder 420 receives a feature map of image features from the image encoder 410 and iteratively decodes the feature map through subsequent spatial dimensions, the skip connections provide the residual image features concatenated with the depth features into the respective iterations.”); this shows depth decoder receiving feature maps to iteratively decode and is for generating depth map which would be the aforementioned layered depth map from McCready; and iteratively generating an intermediate layered depth map based on the decoded plurality of feature maps (Guizilini paragraph 39 teaches “iteratively decodes the feature map through subsequent spatial dimensions, the skip connections provide the residual image features concatenated with the depth features into the respective iterations. In this way, the depth model 250 injects the sparse depth data 440 into the encoder/decoder structure and improves an accuracy of the depth map 260 as the final output” and paragraph 47 teaches “the image decoder 420 functions includes unfolding (i.e., adapting dimensions of the tensor to extract the features) the previously encoded spatial information in order to derive the depth map 260 according to learned correlations associated with the encoded features. That is, the decoding layers generally function to up-sample, through sub-pixel convolutions and other mechanisms, the previously encoded features into the depth map 260. In one or more arrangements, the decoding layers comprise unpacking blocks, two-dimensional convolutional layers, and inverse depth layers that function as output layers for different spatial scales”); relying on previously encoded features in depth map and decoding layers comprising inverse depth layers that function as output for different scales shows the aforementioned iteration would generate intermediate layered depth map (the earlier generated ones) and this is based on the aforementioned feature maps since depth map is generated from such. Guizilini is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of efficiently and iteratively decoding for the generating of depth map from feature maps. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Sagong and McCready with the iteratively decoding for generation of depth map techniques of Guizilini to improve the accuracy of the depth map (Guizilini,paragraph 56). This would be done by iterative technique applied to generation and decoding.
Regarding claim 5, the combination of Sagong, McCready and Guizilini teaches wherein two or more of the decoded plurality of feature maps have different volumetric dimensions (Guizilini, paragraph 42 teaches “assuming a batch size of 1 and disregard the batch index, an input W×H×1 depth map {tilde over (D)} is sparsified by the sparsification layer by gathering valid pixels” and paragraph 46 teaches “output of the image encoder 410 is, in one approach, a feature map having a particular dimension (e.g., 512×H/32×W/32) that is transformed in relation to the image 450 (e.g., 3×H×W)”); this shows different representations with different volumetric dimensions which means that one of ordinary skill in the art would understand the volumetric dimensions of feature maps that are decoded would be unique/different in certain scenarios (also, decoder operating at different scales [such as example of feature map particular dimensions which varies] means feature maps can have different dimensions at different stages). The same motivations used in claim 4 apply here in claim 5.
Regarding claim 14, the apparatus claim 14 recites similar limitations as method claim 4, and thus is rejected under similar rationale.
Claim(s) 7 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Sagong and McCready as applied to claims 1 and 11 above, and further in view of Hui et al. (Depth Map Super-Resolution by Deep Multi-Scale Guidance), hereinafter referenced as Hui.
Regarding claim 7, the combination of Sagong and McCready fails to teach wherein generating the layered depth map includes iteratively refining the layered depth map from a low resolution to a high resolution while reducing a number of layers associated with the layered depth map.
However, Hui teaches wherein generating the layered depth map includes iteratively refining the layered depth map from a low resolution to a high resolution (Hui, page 354, fig. 2 description teaches “Over-texture transfer in depth map refinement and upsampling using intensity guidance” and page 362, third paragraph teaches “trained each MS-Net and MSG-Net for 5e+5 iterations”); trained over multiple iterations shows the depth map generation includes iterative refinement and the upsampling shows from low to high resolution; while reducing a number of layers associated with the layered depth map (Hui, page 358 last two paragraph teach “Suppose the upscaling factor is s =2m, then there are M layers… M =3(m+1)” and page 361, first paragraph teaches “For an upscaling factor s =2m, there are only (m + 2) layers. MS-Net can be expressed as follows”); this shows reduced number of layers that are associated with the aforementioned layer depth map (3m+1 layers versus m+2 layers) from upsampling. Hui is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of upsampling using layered depth representation to increase resolution in images. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Sagong and McCready with the upsampling techniques of Hui to achieve optimal upsampling and improve the quality of the recovered HR image (Hui, page 357, section 3, last two paragraphs). This leads to better user experience.
Regarding claim 16, the apparatus claim 16 recites similar limitations as method claim 7, and thus is rejected under similar rationale.
Claim(s) 8 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Sagong and McCready as applied to claims 1 and 11 above, and further in view of Zitnick, III et al. (U.S. Patent Application Publication No. 2005/0286759), hereinafter referenced as Zitnick.
Regarding claim 8, the combination of Sagong and McCready fails to explicitly teach wherein generating the image includes blending the plurality of images using a weight associated with the layered depth map.
However, Zitnick explicitly teaches wherein generating the image includes blending the plurality of images using a weight associated with the layered depth map. (Zitnick, paragraph 48 teaches “Each view is rendered independently by warping each of its two layers using the associated depth map and compositing the boundary layer over the main layer. The two warped views are then blended based on their proximity to the novel view”); this shows image/view rendering/generation by warping and blending which occurs based on proximity/weight and this weight/proximity is associated with layered depth map because depth maps are used to create the images being blended. Zitnick is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of blending images and weighting associated with layered depth map. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Sagong and McCready with the blending and weighting techniques of Zitnick to improve the quality of the interpolated views (Zitnick, paragraph 44). This is due to multiple images blended with weights by giving certain features that would improve quality/accuracy more weight.
Regarding claim 17, the apparatus claim 17 recites similar limitations as method claim 8, and thus is rejected under similar rationale.
Claim(s) 9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Sagong and McCready as applied to claims 1 and 11 above, and further in view of Bumpas et al. (U.S. Patent Application Publication No. 2019/0278994), hereinafter referenced as Bumpas.
Regarding claim 9, the combination of Sagong and McCready fails to explicitly teach wherein the layered depth map includes a plurality of layers with spatial dimensions including the view perspective.
However, Bumpas explicitly teaches wherein the layered depth map includes a plurality of layers with spatial dimensions including the view perspective (Bumpas, paragraph 39 teaches “Each convolutional layer 202 may include a fixed-size feature map that can be represented as a 3-dimensional matrix having dimensions W′×H'×D', where D′ corresponds to the number of layers (or “depth”) within that feature map”); this shows spatial dimensions W’xH’ included in layers of depth map and this would include the view perspective from Sagong when viewed in combination because as aforementioned Sagong paragraph 62 has the view perspective which provides the capabilities of viewing from the view perspective. Bumpas is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of layers in depth map having spatial dimensions. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Sagong and McCready with the spatial dimensions techniques of Bumpas to ensure much better accuracy (Bumpas, paragraph 67). This would be due to the specific dimensions.
Regarding claim 18, the apparatus claim 18 recites similar limitations as method claim 9, and thus is rejected under similar rationale.
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over as the combination of Sagong, McCready and Guizilini applied to claims 4 and 14 above, and further in view of Hui.
Regarding claim 6, the combination of Sagong, McCready and Guizilini fails to teach wherein generating the layered depth map includes upsampling the intermediate layered depth map, and activating the upsampled layered depth map using a non-linear activation function.
However, Hui teaches wherein generating the layered depth map includes upsampling the intermediate layered depth map, (Hui, page 357, section 3 title teaches “Intensity-Guided Depth Map Upsampling”, and section 3.1 first paragraph teaches “upsample a LR image Dl not in a single level but progressively in multiple levels to a desired HR image Dh”); upsampling depth map “D” progressively in multiple levels shows the intermediate layered depth map from the above combination (aforementioned in Guizilini paragraphs 39 and 47) would also be upsampled for generation of layered depth map; and activating the upsampled layered depth map using a non-linear activation function (Hui, page 359, first paragraph teaches “Each layer is followed by an activation function for non-linear mapping except the top layer”); if each layer is followed by activating using non-linear activation function that means the aforementioned upsampled layer is as well.
Hui is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of upsampling using layered depth representation to increase resolution in images. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Sagong and McCready with the upsampling techniques of Hui to achieve optimal upsampling and improve the quality of the recovered HR image (Hui, page 357, section 3, last two paragraphs). This leads to better user experience.
Regarding claim 15, apparatus claim 15 recites similar limitations as method claim 5 and 6, and thus is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen et al. (U.S. Patent Application Publication No. 2025/0022153), abstract teaches “feature pyramids can be used to compute a respective cost volume for each feature pyramid at each level of resolution… depth map for each input image can be generated” and paragraph 41 teaches “all of the BN layers and nonlinear activations”; this shows the concept of feature pyramids at different resolutions, depth maps generated and non-linear activation function.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.U.A./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
