DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3 June 2026 has been entered.
Status of Claims
Applicant's amendments filed on 3 June 2026 have been entered. Claims 1 and 7 have been amended. No claims have been canceled. No claims have been added. Claims 1-20 are still pending in this application, with claims 1, 7, 13, and 17 being independent.
Response to Arguments
Applicant's arguments filed 3 June 2026 have been fully considered but they are not persuasive.
With respect to claims 13 and 17, Applicant argues that “Liu does not describe applying an integrated conversion to a normal map for a texture used to render a three-dimensional model in a virtual scene, nor does it disclose compressing such an integrated image using a compression method,” alleging that “Paragraph [0022] of Liu describes a conventional method for detecting planar surfaces in depth data that involves using a planar equation to generate plane hypotheses, wherein the parameters for the equation are first converted into a Polar coordinate system representation, the parameter space is quantized, and the system searches the parameter space to generate results.” Examiner respectfully disagrees, noting that the claimed limitation merely requires “applying an integrating conversion to a normal map, wherein each pixel in the integrated image is represented as floating-point numbers,” which is precisely what is accomplished in this cited portion of Liu, e.g.: “the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh)…where δ is a threshold to account for depth noise. Since {ah, bh, ch, dh} are floating point numbers.” Thus, Examiner maintains the rejections of independent claims 13 and 17 for at least the above reasons.
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on every reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-11, and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (US Pub. 2017/0039731), hereinafter Liu, in view of Venkataraman et al. (US Patent 12340538), hereinafter Venkataraman, and further in view of Prasad et al. (US Pub. 2023/0085156), hereinafter Prasad.
Regarding claim 1, Liu discloses a computer-implemented method of compressing normal maps using an integrating conversion, the method comprising: applying an integrating conversion to a normal map, wherein each pixel in the integrated image is represented as floating-point numbers (Paragraph [0022]: where (xp, yp, zp) are the coordinates in the units (such as meters) of the content (or object or feature point) in the image and for every pixel or point in the image. To perform the conventional methods, the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh)…where δ is a threshold to account for depth noise. Since {ah, bh, ch, dh} are floating point numbers, in practice, the parameter space is quantized with large bin size to reduce the search time, but even with quantization, the number of hypotheses is still large); and compressing the integrated image using a compression method (Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor. A coder 1032, which may be an encoder, decoder, or both, also may be provided. As an encoder 1032 and antenna 1034 may be provided to compress the modified image date for transmission to other devices that may display or store the image. It will be understood that the image processing system 1000 also may include a decoder (or encoder 1032 may include a decoder) to receive and decode image data for processing by the system 1000. Otherwise, the processed image 1030 may be displayed on display 1028 or stored in memory 1024. As illustrated, any of these components may be capable of communication with one another and/or communication with portions of logic modules 1004 and/or imaging device 1002. Thus, processors 1020 may be communicatively coupled to both the image device 1002 and the logic modules 1004 for operating those components. By one approach, although image processing system 1000, as shown in FIG. 10, may include one particular set of blocks or actions associated with particular components or modules, these blocks or actions may be associated with different components or modules than the particular component or module illustrated here).
Liu does not explicitly disclose obtaining a normal map for a texture used to render a three-dimensional model in a virtual scene; or wherein applying the integrating conversion to the normal map includes determining a slope of a projection of a normal vector from a first point of the normal map and using the slope to determine a value for the integrated image corresponding to a pixel at a second point in the normal map.
However, Venkataraman teaches rendering of models including texturing (Abstract; Column 11), further comprising obtaining a normal map for a texture used to render a three-dimensional model in a virtual scene (Fig. 10; Column 37, lines 4-18: a neural network trained according to embodiments of the present disclosure would disambiguate between the different noisy or inconsistent surface normals N computed from the different views, modalities, and spectral information captured (e.g., where the surface normals maps N may differ in accordance with the noise or variability of the polarization signature due to differences in albedo, wavelength, and texture copy artifacts due to viewing direction, as well as other noise in ambiguity in the image capture process). Comparing the shape estimates from the model 960 against the shape estimates computed directly from the images 980 in a comparison module 990 show that a properly-trained computer vision model 940 produces smoother and more accurate shape estimates than comparative techniques; Column 38, lines 7-39: FIG. 10 is a schematic block diagram depicting a computer vision model according to some embodiments of the present disclosure. As shown in FIG. 10, in some embodiments, the computer vision model 1040 includes a first neural network 1041 (e.g., using an architecture as described above) that is trained to compute shape estimates 1060 (e.g., surface normals maps) from irradiance images 1050 captured from a scene. The computer vision model 1040 in these embodiments further include a second neural network 1042 (e.g., a convolutional neural network) or a shape-to-irradiances model 1042 that is trained to map a computed surface normals map (e.g., trained using the ground truth labels in the dataset) to synthesized image irradiances 1070 from polarization filtered cameras (e.g., estimates of what the captured polarization raw frames would give rise to the surface normals map); and wherein applying the integrating conversion to the normal map includes determining a slope of a projection of a normal vector from a first point of the normal map and using the slope to determine a value for the integrated image corresponding to a pixel at a second point in the normal map (Column 19, lines 52-67: some aspects of embodiments of the present disclosure relate to applying SfP theory to detect or measure the gradients of surfaces (e.g., the orientation of surfaces or their surface normals or directions perpendicular to the surfaces) based on the raw polarization frames of the objects, as captured by the polarization cameras among the main camera 10 and the support cameras 30. Computing these gradients produces a gradient map (or slope map or surface normals map) identifying the slope of the surface depicted at each pixel in the gradient map. These gradient maps can then be used when estimating the shape and/or pose of the object by supplying these gradient maps or surface normals maps to a trained computer vision model (e.g., a convolutional neural network) and/or by aligning a pre-existing 3-D model (e.g., CAD model) of the object with the measured surface normals (gradients or slopes) of the object in based on the slopes of the surfaces of the 3-D model, as described in more detail below; Column 34, line 46-Column 35, line 7: rendered shape estimates may include images of a virtual scene with one or more 3-D models posed in accordance with the estimated poses of the corresponding objects depicted in the scene, and the images are rendered from the perspective of virtual cameras having intrinsic and extrinsic camera parameters matching those of the main camera 10 and/or support cameras 30, such that each rendered view corresponds to a view from the observed images 18. For example, rendering color images of a scene may provide estimates regarding the outline or silhouette of the object as viewed from a particular camera. As another example, rendering a depth map based on the 3-D model can provide a high resolution depth map or point cloud of the shape of the objects in the scene with substantially no noise compared to point clouds or depth maps generated from depth camera systems. As a third example, rendering a surface normals map of the surface normals directions of the surfaces of the 3-D model produces a higher resolution surface normals map with substantially no noise compared to surface normals maps computed directly from depth maps from depth camera systems or computed from polarization raw frames based on shape from polarization techniques. In various embodiments, these surface normals maps may be rendered by directly detecting the surface normals (e.g., the angles or slopes or orientations) of the surfaces of the posed 3-D model, or may be computed from the high resolution depth map rendered from the 3-D model). Venkataraman teaches that this will allow for accuracy of models and scenes (Column 6). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Liu with the features above as taught by Venkataraman so as to allow for enhanced realism as presented by Venkataraman.
While Liu teaches applying the integrating conversion (Paragraph [0022]: where (xp, yp, zp) are the coordinates in the units (such as meters) of the content (or object or feature point) in the image and for every pixel or point in the image. To perform the conventional methods, the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh)), Liu, in view of Venkataraman does not explicitly disclose wherein applying the integrating conversion to the normal map produces an integrated image of the normal map.
However, Prasad teaches image compression (Paragraphs [0021]-[0024]; Paragraphs [0052]-[0058]), further comprising wherein applying the integrating conversion to the normal map produces an integrated image of the normal map (Fig. 2E; Paragraph [0029]: one or more of the frame 220D and the frame 220E—e.g., corresponding to a depth map and a surface normal map, respectively—may be used to generate the edge map of frame 220F. In other embodiments, the frame 220C—without first determining depth and/or surface normal information—may be used to generate the edge map of frame 220F. For example, changes in pixel values beyond a threshold amount may indicate edge locations, and this information may be used to determine the edges for the edge map. However, using the pixel values alone without the depth map and/or the surface normal map may lead to less accurate results than using the depth map and/or the surface normal map). Prasad teaches that this will allow for increasingly accurate results (Paragraphs [0029]-[0031]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Liu, in view of Venkataraman with the features wherein applying the integrating conversion to the normal map produces an integrated image of the normal map as taught by Prasad so as to provide accurate results as presented by Prasad.
Regarding claim 2, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 1, Liu discloses wherein the normal map comprises vectors at each fragment that are normalized in three-dimensional space (Paragraph [0022]: where (xp, yp, zp) are the coordinates in the units (such as meters) of the content (or object or feature point) in the image and for every pixel or point in the image. To perform the conventional methods, the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh); Paragraph [0044]: the plane hypothesis parameters in the plane hypothesis equation may be determined by finding the normal of the plane. For example, the three sample points forming the triangular area or plane first may be used to determine two vectors forming the plane both stemming from the main sample point. Thus, sample points 502 and 504 may form a first vector {right arrow over (a)}. while sample points 502 and 510 form a second vector {right arrow over (b)}. The three dimensional points of each of the three sample points then may be used to determine the normal of the plane by finding the cross product of the two vectors where: where Ns is a Normal vector of the plane hypothesis based on the sample points, and continuing with the example of plane 534 on image 500, where ps is the main sample point).
Regarding claim 3, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 1, Prasad discloses wherein applying the integrating conversion to the normal map comprises optimizing per-pixel difference between the normal map and a map obtained when a differential operator is applied to each pixel and its surrounding pixels (Paragraph [0023]: the entropy loss function may measure pixel gradients within portions of the pre-filtered frame—except for portions corresponding to edges as identified using the edge maps—in order to reduce the gradient between neighboring pixels. In such an example, the higher the entropy control parameter value, the more a higher gradient is penalized using the entropy loss function. As such, where a high value for the entropy control parameter is used (e.g., indicative of lower frame entropy), the gradient between neighboring or surrounding pixels may be reduced such that differences in pixel values between neighboring or surrounding pixels are minimal. Similarly, for lower entropy control parameter values (e.g., indicative of higher frame entropy), the gradient between neighboring or surrounding pixels may be reduced less such that differences in pixels are allowed to be greater (but not as great as a full entropy frame). Where more than one loss function is used, the loss functions may be weighted during training; Paragraph [0037]: the entropy loss function may measure pixel gradients within portions of the pre-filtered frame 112—except for portions corresponding to edges as identified using the edge maps 106—in order to reduce the gradient between neighboring pixels. In such an example, the higher the entropy control parameter value, the more a higher gradient is penalized using the entropy loss function. As such, where a high value for the entropy control parameter is used (e.g., indicative of lower frame entropy), the gradient between neighboring or surrounding pixels may be reduced such that differences in pixel values between neighboring or surrounding pixels are minimal. Similarly, for lower entropy control parameter values (e.g., indicative of higher frame entropy), the gradient between neighboring or surrounding pixels may be reduced less such that differences in pixels are allowed to be greater (but not as great as a full entropy frame); Paragraph [0044]: method 300, at block B308, includes computing, using a second loss function, a second loss value based at least in part on comparing pixel values of proximately located pixels within the pre-filtered frame. For example, the entropy loss function may measure pixel gradients within portions of the pre-filtered frame—except for portions corresponding to edges as identified using the edge maps—in order to reduce the gradient between neighboring pixels. For example, for a DNN 110 corresponding to a higher entropy control parameter (and thus a lower frame entropy), differences between the neighboring pixels may be penalized more. As another example, for a DNN 110 corresponding to a lower entropy control parameter (and thus a higher frame entropy), differences between the neighboring pixels may be penalized less—but still penalized in order to reduce the frame entropy from that of the original frame).
Regarding claim 4, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 1, Liu discloses the method further comprising applying quantization to the integrated image, wherein applying quantization to the integrated image produces an RGB integrated image of the normal map, wherein the integrated image compressed using the compression method comprises the RGB integrated image (Paragraphs [0026]-[0038]: guided by the given depth image, the present method samples the plane parameter space in an educated manner, and hence achieves a much higher frame rate (about 300 fps at video graphics array (VGA) depth) than the conventional methods. Since the present method also includes gravity-sensitive (such as an inertial measurement unit (IMU)) sensor data incorporation to further improve the frame rate when only planes of specific orientation (e.g. horizontal planes, such as ground) are needed, a different voting scheme that takes into account local geometry to improve the accuracy at plane boundaries, and a color-based refinement stage to fill holes and unfit areas, this results in a method and system of plane detection that may be hundreds of times faster than the known methods. The current method also produces a smoother boundary of the planes with the incorporation of the local geometry and color information versus that of the known methods… operations also may be performed to further eliminate plane hypotheses and refine the quality of the planes before providing a final index map of the plane hypotheses. This may include suppression of the non-largest planes when planes overlap, and color-based hole and gap filling using region erosion and region growth as described in detail below… the sample points are used to compute the plane hypotheses. By forming a plane hypothesis for each sample point rather than for each bin in the quantized parameter space, the computational load for detecting the planar surfaces in the image is reduced significantly resulting in a large increase in frame rate during the detection process; Paragraph [0086]: such technology may include a camera such as a digital camera system, a dedicated camera device, or an imaging phone or tablet, whether a still picture or video camera, camera that provides a preview screen, or some combination of these. Thus, in one form, imaging device 1002 may include camera hardware and optics including one or more sensors as well as auto-focus, zoom, aperture, ND-filter, auto-exposure, flash, and actuator controls. These controls may be part of a sensor module or component for operating the sensor that can be used to generate images for a viewfinder and take still pictures or video. The imaging device 1002 also may have a lens, an image sensor with a RGB Bayer color filter, an analog amplifier, an A/D converter, other components to convert incident light into a digital signal, the like, and/or combinations thereof. The digital signal also may be referred to as the raw image data herein).
Regarding claim 5, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 4, Liu discloses wherein a map obtained after applying quantization to the integrated image corresponds to a height map of the normal map (Paragraphs [0181]-[0184]: instead of the depth difference, the derivate of the depth gradient is used to determine the interpolation function by the decoder. This could either be the 1D derivate for x and y direction (performed individually as z/dx and z/dy) or the 2D-derviate to perform the interpolation in one step (z/dxdy). Such an approach can represent non-linear depth gradients better than the simple difference approach…where the depth information in the geometry does not follow linear quantization, the same quantization is applied for determining the new depth values for the interpolated 3D points…the depth interpolation weighting is derived from a 2D gaussian filter centred at the desired reconstruction coordinate in the geometry picture (spatial kernel), taking only valid values into account…instead of the depth weighted interpolation, a depth-guided bilateral filter is used for interpolating new geometry values, where depth z (geometry image) guides the range kernel (intensity) and the spatial kernel is based on the coordinate differences in the geometry image).
Regarding claim 7, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the one or more processors, which are disclosed by Liu, Fig. 10; Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor); thus they are rejected on similar grounds.
Regarding claim 8, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 9, the limitations of this claim substantially correspond to the limitations of claim 3; thus they are rejected on similar grounds.
Regarding claim 10, the limitations of this claim substantially correspond to the limitations of claim 4; thus they are rejected on similar grounds.
Regarding claim 11, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Regarding claim 13, Liu discloses a computer-implemented method of improving rendering of three-dimensional models by decompressing texture maps compressed using an integrated conversion (Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor. A coder 1032, which may be an encoder, decoder, or both, also may be provided. As an encoder 1032 and antenna 1034 may be provided to compress the modified image date for transmission to other devices that may display or store the image. It will be understood that the image processing system 1000 also may include a decoder (or encoder 1032 may include a decoder) to receive and decode image data for processing by the system 1000. Otherwise, the processed image 1030 may be displayed on display 1028 or stored in memory 1024. As illustrated, any of these components may be capable of communication with one another and/or communication with portions of logic modules 1004 and/or imaging device 1002. Thus, processors 1020 may be communicatively coupled to both the image device 1002 and the logic modules 1004 for operating those components. By one approach, although image processing system 1000, as shown in FIG. 10, may include one particular set of blocks or actions associated with particular components or modules, these blocks or actions may be associated with different components or modules than the particular component or module illustrated here); decompressing a compressed integrated image corresponding to a normal map (Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor. A coder 1032, which may be an encoder, decoder, or both, also may be provided. As an encoder 1032 and antenna 1034 may be provided to compress the modified image date for transmission to other devices that may display or store the image. It will be understood that the image processing system 1000 also may include a decoder (or encoder 1032 may include a decoder) to receive and decode image data for processing by the system 1000. Otherwise, the processed image 1030 may be displayed on display 1028 or stored in memory 1024. As illustrated, any of these components may be capable of communication with one another and/or communication with portions of logic modules 1004 and/or imaging device 1002. Thus, processors 1020 may be communicatively coupled to both the image device 1002 and the logic modules 1004 for operating those components. By one approach, although image processing system 1000, as shown in FIG. 10, may include one particular set of blocks or actions associated with particular components or modules, these blocks or actions may be associated with different components or modules than the particular component or module illustrated here).
Liu does not explicitly disclose a normal map for a texture used to render a three-dimensional model in a virtual scene.
However, Venkataraman teaches rendering of models including texturing (Abstract; Column 11), further comprising a normal map for a texture used to render a three-dimensional model in a virtual scene (Fig. 10; Column 37, lines 4-18: a neural network trained according to embodiments of the present disclosure would disambiguate between the different noisy or inconsistent surface normals N computed from the different views, modalities, and spectral information captured (e.g., where the surface normals maps N may differ in accordance with the noise or variability of the polarization signature due to differences in albedo, wavelength, and texture copy artifacts due to viewing direction, as well as other noise in ambiguity in the image capture process). Comparing the shape estimates from the model 960 against the shape estimates computed directly from the images 980 in a comparison module 990 show that a properly-trained computer vision model 940 produces smoother and more accurate shape estimates than comparative techniques; Column 38, lines 7-39: FIG. 10 is a schematic block diagram depicting a computer vision model according to some embodiments of the present disclosure. As shown in FIG. 10, in some embodiments, the computer vision model 1040 includes a first neural network 1041 (e.g., using an architecture as described above) that is trained to compute shape estimates 1060 (e.g., surface normals maps) from irradiance images 1050 captured from a scene. The computer vision model 1040 in these embodiments further include a second neural network 1042 (e.g., a convolutional neural network) or a shape-to-irradiances model 1042 that is trained to map a computed surface normals map (e.g., trained using the ground truth labels in the dataset) to synthesized image irradiances 1070 from polarization filtered cameras (e.g., estimates of what the captured polarization raw frames would give rise to the surface normals map). Venkataraman teaches that this will allow for accuracy of models and scenes (Column 6). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Liu with the features above as taught by Venkataraman so as to allow for enhanced realism as presented by Venkataraman.
While Liu teaches decompressing maps compressed using an integrated conversion (Paragraph [0022]: where (xp, yp, zp) are the coordinates in the units (such as meters) of the content (or object or feature point) in the image and for every pixel or point in the image. To perform the conventional methods, the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh)), Liu does not explicitly disclose decompressing texture maps compressed using an integrated conversion; or applying a differential conversion to the decompressed integrated image, wherein an image obtained after applying the differential conversion to the decompressed integrated image comprises the normal map.
However, Prasad teaches image compression (Paragraphs [0021]-[0024]; Paragraphs [0052]-[0058]), further comprising wherein applying the integrating conversion to the normal map produces an integrated image of the normal map (Fig. 1B; Paragraph [0029]: one or more of the frame 220D and the frame 220E—e.g., corresponding to a depth map and a surface normal map, respectively—may be used to generate the edge map of frame 220F. In other embodiments, the frame 220C—without first determining depth and/or surface normal information—may be used to generate the edge map of frame 220F. For example, changes in pixel values beyond a threshold amount may indicate edge locations, and this information may be used to determine the edges for the edge map. However, using the pixel values alone without the depth map and/or the surface normal map may lead to less accurate results than using the depth map and/or the surface normal map; Paragraph [0051]: DNN(s) 430 may store several DNNs, each corresponding to different levels of filtering—e.g., to a different entropy control parameter. Depending on the level of filtering, a pre-filtered frame may include more or less visual detail when displayed on the display 408 of the end-user device 420. Turning briefly to FIGS. 5A-5D, for example, each of frames 500A/B/C/D correspond to a different entropy control parameters (e.g., 0.1, 0.5, 1.0, and 3.0, respectively), which correspond to high entropy, medium entropy, low entropy, and very low entropy, respectively. Each of the frames 500A/B/C/D include an archway 502A/B/C/D. As can be seen, archway 502A includes a significant level of detail with well-defined and visible bricks in the archway 502A. Archway 502B includes less detail when compared to 502A. Archway 502B includes some degree of texture to illustrate bricks, but they are not well defined visually. Archway 502C includes less detail when compared to 502B. The archway 502C includes some degree of detail for the archway 502C, but no bricks are visible. Lastly, archway 502D includes still less detail when compared to archway 502C. The archway 502D includes no texture and no bricks are visible. However, the edges of the archway 502D are maintained, which allows a user to navigate a game corresponding to the frames 500A/B/C/D); and applying a differential conversion to the decompressed integrated image, wherein an image obtained after applying the differential conversion to the decompressed integrated image comprises the normal map (Paragraph [0023]: the entropy loss function may measure pixel gradients within portions of the pre-filtered frame—except for portions corresponding to edges as identified using the edge maps—in order to reduce the gradient between neighboring pixels. In such an example, the higher the entropy control parameter value, the more a higher gradient is penalized using the entropy loss function. As such, where a high value for the entropy control parameter is used (e.g., indicative of lower frame entropy), the gradient between neighboring or surrounding pixels may be reduced such that differences in pixel values between neighboring or surrounding pixels are minimal. Similarly, for lower entropy control parameter values (e.g., indicative of higher frame entropy), the gradient between neighboring or surrounding pixels may be reduced less such that differences in pixels are allowed to be greater (but not as great as a full entropy frame). Where more than one loss function is used, the loss functions may be weighted during training). Prasad teaches that this will allow for increasingly accurate results (Paragraphs [0029]-[0031]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Liu with the features of wherein applying the integrating conversion to the normal map produces an integrated image of the normal map; and applying a differential conversion to the decompressed integrated image, wherein an image obtained after applying the differential conversion to the decompressed integrated image comprises the normal map as taught by Prasad so as to provide accurate results as presented by Prasad.
Regarding claim 14, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 13, Liu discloses wherein the normal map comprises vectors at each fragment that are normalized in three-dimensional space (Paragraph [0022]: where (xp, yp, zp) are the coordinates in the units (such as meters) of the content (or object or feature point) in the image and for every pixel or point in the image. To perform the conventional methods, the parameters for the equation (1) h(ah, bh, ch, dh) is first converted into a Polar coordinate system representation: h(ah, bh, ch, dh)=h(cos θ sin φ, sin θ sin φ, cos φ, dh), the parameter space (θ, φ, dh) is quantized, and the system effectively searches the parameter space for every possible combination of (θ, φ, dh), where θ is the angle of the normal vector (ah, bh, ch) on the xy plane and φ is the angle between the xy plane and the normal vector (ah, bh, ch) in z direction. θ ranges from 0 to 360 degrees and φ ranges from 0 to 180 degrees. Assuming the target application operates within a 5 meter range, and hence dh ranges from 0 to 5, 5 degrees is used for both θ and φ, then 0.05 meters for dh to have reasonable plane detection results. The final plane hypotheses are those that have sufficient pixel support. A pixel p(xp, yp, zp) supports a plane hypothesis: h(ah, bh, ch, dh); Paragraph [0044]: the plane hypothesis parameters in the plane hypothesis equation may be determined by finding the normal of the plane. For example, the three sample points forming the triangular area or plane first may be used to determine two vectors forming the plane both stemming from the main sample point. Thus, sample points 502 and 504 may form a first vector {right arrow over (a)}. while sample points 502 and 510 form a second vector {right arrow over (b)}. The three dimensional points of each of the three sample points then may be used to determine the normal of the plane by finding the cross product of the two vectors where: where Ns is a Normal vector of the plane hypothesis based on the sample points, and continuing with the example of plane 534 on image 500, where ps is the main sample point).
Regarding claim 15, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 13, Prasad discloses wherein the compressed integrated image is decompressed using a decompression method complimentary to a compression method used to compress the compressed integrated image (Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor. A coder 1032, which may be an encoder, decoder, or both, also may be provided. As an encoder 1032 and antenna 1034 may be provided to compress the modified image date for transmission to other devices that may display or store the image. It will be understood that the image processing system 1000 also may include a decoder (or encoder 1032 may include a decoder) to receive and decode image data for processing by the system 1000. Otherwise, the processed image 1030 may be displayed on display 1028 or stored in memory 1024. As illustrated, any of these components may be capable of communication with one another and/or communication with portions of logic modules 1004 and/or imaging device 1002. Thus, processors 1020 may be communicatively coupled to both the image device 1002 and the logic modules 1004 for operating those components. By one approach, although image processing system 1000, as shown in FIG. 10, may include one particular set of blocks or actions associated with particular components or modules, these blocks or actions may be associated with different components or modules than the particular component or module illustrated here).
Regarding claim 16, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 13, the method further comprising converting the decompressed integrated image into a floating-point representation of the normal map and applying the differential conversion to the floating-point representation of the normal map, wherein converting the decompressed integrated image into a floating-point representation comprises: multiplying each point in the normal map by a scale parameter (Prasad: Paragraph [0023]: the entropy loss function may measure pixel gradients within portions of the pre-filtered frame—except for portions corresponding to edges as identified using the edge maps—in order to reduce the gradient between neighboring pixels. In such an example, the higher the entropy control parameter value, the more a higher gradient is penalized using the entropy loss function. As such, where a high value for the entropy control parameter is used (e.g., indicative of lower frame entropy), the gradient between neighboring or surrounding pixels may be reduced such that differences in pixel values between neighboring or surrounding pixels are minimal. Similarly, for lower entropy control parameter values (e.g., indicative of higher frame entropy), the gradient between neighboring or surrounding pixels may be reduced less such that differences in pixels are allowed to be greater (but not as great as a full entropy frame). Where more than one loss function is used, the loss functions may be weighted during training); and assigning each pixel a value based on a value map generated by applying quantization to an integrated image of the normal map prior to compression (Liu: Paragraphs [0181]-[0184]: instead of the depth difference, the derivate of the depth gradient is used to determine the interpolation function by the decoder. This could either be the 1D derivate for x and y direction (performed individually as z/dx and z/dy) or the 2D-derviate to perform the interpolation in one step (z/dxdy). Such an approach can represent non-linear depth gradients better than the simple difference approach…where the depth information in the geometry does not follow linear quantization, the same quantization is applied for determining the new depth values for the interpolated 3D points…the depth interpolation weighting is derived from a 2D gaussian filter centred at the desired reconstruction coordinate in the geometry picture (spatial kernel), taking only valid values into account…instead of the depth weighted interpolation, a depth-guided bilateral filter is used for interpolating new geometry values, where depth z (geometry image) guides the range kernel (intensity) and the spatial kernel is based on the coordinate differences in the geometry image).
Regarding claim 17, the limitations of this claim substantially correspond to the limitations of claim 13 (except for the one or more processors, which are disclosed by Liu, Fig. 10; Paragraph [0089]: image processing system 1000 may have one or more processors 1020 which may include a dedicated image signal processor (ISP) 1022 such as the Intel Atom, memory stores 1024, one or more displays 1028 to provide images 1030, a coder 1032, and antenna 1026. In one example implementation, the image processing system 1000 may have the display 1028, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor); thus they are rejected on similar grounds.
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 14; thus they are rejected on similar grounds.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 15; thus they are rejected on similar grounds.
Regarding claim 20, the limitations of this claim substantially correspond to the limitations of claim 16; thus they are rejected on similar grounds.
Claims 6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, in view of Venkataraman, in view of Prasad, and further in view of Cohen-Tidhar et al. (US Pub. 2023/0177734), hereinafter Cohen-Tidhar.
Regarding claim 6, Liu, in view of Venkataraman, and further in view of Prasad teaches the computer-implemented method of claim 1.
Liu, in view of Venkataraman, and further in view of Prasad does not explicitly disclose wherein the compression method comprises a compression method associated with the AVIF file format or a method associated with the JPEG XL file format.
However, Cohen-Tidhar teaches image compression (Paragraph [0067]), further comprising wherein the compression method comprises a compression method associated with the AVIF file format or a method associated with the JPEG XL file format (Paragraph [0067]: Efficiency Image File Format (HEIF) file or container, or other type of a single file or a single container that can support two or more encoding (or compression) techniques that are implemented with regard to content-portions or content-layers of a single image. Some embodiments may thus operate to perform image encoding in a content-aware manner, by selecting the most suitable single media format that would store the composite image, and by compositing a plurality of encoding formats that encode image-portions or image-layers of a single image into a single container (e.g., SVG or PDF or HEIF, or other container that can contain multiple payload codecs or multiple payload-contents that are encoded separately by two or more different encoding techniques), and by selecting the coding techniques and parameters per image-region within an expressive format that can achieve the most efficient performance goals for each content type (e.g., to select an encoding technique that supports macroblocks or layering, such as JXL JPEG XL encoding, or AVIF or AV1 encoding). Cohen-Tidhar teaches that this will enable achieving the most efficient performance goals in compression (Paragraph [0067]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Liu, in view of Venkataraman, and further in view of Prasad with the features of above as taught by Cohen-Tidhar so as to provide efficient performance as presented by Cohen-Tidhar.
Regarding claim 12, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights