Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This communication is in response to the action filed on 03/27/2026.
Claims 1, 11, 16, and 21 have been amended. Claim 6 is canceled. Claims 1-5, and 7-22 are pending.
Response to Arguments
Applicant’s arguments filed on 03/27/2026 on pages 8-10, under REMARKS with respect to 35 U.S.C. 103 claim rejections to claims 1-5, and 7-22 have been fully considered and are persuasive. The rejections to the claims have been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of US 2012/0033872 A1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-3, 5, 7, 11-12, 14, 16-17, 19-22 are rejected under 35 § U.S.C. 103 as being obvious over US 2012/0033872 A1 to CHO et al. (hereinafter “CHO”), in view of US 2016/0330434 A1 to CHEN (hereinafter “CHEN”).
As per claim 1, CHO discloses a method (a computing system to perform image processing methods related to view extrapolation; abstract; figs 1 and 3), comprising: determining a boundary weight for a target depth pixel of a plurality of depth pixels based on a first gradient magnitude of the target depth pixel in a depth image (the system is adapted to find an object boundary hole acting substantially as a boundary point and has a boundary weight related to a depth pixel of a depth image and is adapted to find the gradient depth value of the specific pixel point; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]) and a different second gradient magnitude of a brightness pixel in a brightness image different from the depth image (the computing system further includes a color image (brightness image) which comprises a brightness/color value related and at the same position as the depth pixel of the depth image but the color image is a different image seen in fig 5; figs 5-6 and 8; paragraphs [0177], [0224-0225]), with the depth image comprising the plurality of depth pixels (the depth image includes depth pixels at specific points x,y and finding a value of depth at the position; fig 6 and 8; paragraphs [0208-0212]), and with the target depth pixel having a first depth value and corresponding to the brightness pixel (each pixel is mapped in the images captured at a specific time T such that the pixel in the same x,y position in the depth image representation and the color image representation is the same pixel in each representation and allows the user to find a first depth value of the position related to the selected brightness pixel position; fig 6, and 8-9; paragraphs [0093], [0126-0127], [0135], [0145], [0174-0175], [0212], [0252], [0257]). CHO fails to disclose determining an energy for the target depth pixel based on the boundary weight; determining a second depth value of the target depth pixel by optimizing the energy; and updating the depth image by assigning the second depth value to the target depth pixel.
CHEN discloses determining an energy for the target depth pixel based on the boundary weight (during step s09 the computing system is adapted to register a first fused depth map with a first and second image comprising target pixels and generate 3D point could data for each pixel/point in each respective image/feature map generated is generated using higher accuracy, and would include an energy value I as the driving current value used to emit light (produce energy); figs 2A-B; paragraphs [0032-0034], [0040-0041]); determining a second depth value of the target depth pixel by optimizing the energy (for example as stated in paragraph 0041 the first preset pixel value is set to 5000 the second preset pixel value is 20,000, the first preset depth value is 10 meters the second preset depth value is 0.5 meters; paragraphs [0032-0034], [0040-0041]); and updating the depth image by assigning the second depth value to the target depth pixel (the aforementioned depth values correspond to different image iterations of the same target pixel, assigning a new depth value to the target pixel as desired based on parameter weighting and generating a map with desired accuracy, wherein the second depth value assigned to the target pixel is 0.5 meters; paragraphs [0032-0034], [0040-0041]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have updating the depth image by assigning the second depth value to the target depth pixel of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 1.
As per claim 2, CHO in view of CHEN discloses the method of claim 1. Modified CHO further discloses wherein: the brightness image and the depth image represent a same object (one pixel may be expressed in color and in depth, the input sequence includes a color sequence representing information about colors of pixels forming a frame and a depth sequence representing information about depth of the pixels and since the pixel may be at the same point in each image the object at that point would also be the same; fig 5; paragraph [0135]), the brightness image comprises a plurality of brightness pixels that includes the brightness pixel (the color image includes a plurality of pixels each comprising their own respective brightness/color values assigned to each individual pixel at its individual point; fig 5; paragraph [0135]), and each respective brightness pixel of the plurality of brightness pixels correspond to a respective depth pixel of the plurality of depth pixels (each respective pixel of the color/brightness image at a specific time T corresponds to a specific position of a depth pixel at the same time T in a depth image; fig 5; paragraph [0135]).
As per claim 3, CHO in view of CHEN discloses the method of claim 1. Modified CHO further discloses wherein the target depth pixel represents a same locus in an object as the brightness pixel (the target depth pixel and the target color/brightness pixel are in the same position x,y of an image captured at a specific time T; fig 5; paragraphs [0135]).
As per claim 5, CHO discloses the method of claim 4. CHO fails to disclose wherein determining the energy for the target depth pixel based on the boundary weight further comprises: determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel.
CHEN discloses wherein determining the energy for the target depth pixel based on the boundary weight further comprises: determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image (the energy value is determined to be enough to power light source however included in the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power (conditional energy) should be supplied to the light source 1, so as to prevent unnecessary power consumption (increase in error); paragraphs [0040-0041], [0046]), wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel (the processor 4 is programmed to control the light source to be driven by a second driving current I2, which is not greater than a minimum of the first driving current I1 (and provided the difference in currents which is the value of uncertainty as it will depend on exact current values of the energy components used to power the light source), to emit light, the first driving current I1 and the second driving current I2 are generated by the processor; paragraphs [0040-0041], [0046]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 5.
As per claim 7, CHO in view of CHEN discloses the method of claim 1. Modified CHO further discloses wherein the depth image and the brightness image are generated based on image data from a same image sensor (the brightness/color image and the depth image are captured by the system using a camera sensor; fig 5; paragraphs [0051], [0142]).
As per claim 11, CHO discloses a camera assembly configured to capture reflected light from at least a portion of the object (the brightness/color image and the depth image are captured by the system using a camera sensor adapted to capture images of an object in a depth image modality and a color/brightness image modality; fig 5; paragraphs [0051], [0142]); and a controller configured to: generate a depth image based on the reflected light (the system is adapted to find an object boundary hole acting substantially as a boundary point and has a boundary weight related to a depth pixel of a depth image and is adapted to find the gradient depth value of the specific pixel point; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]), the depth image comprising a plurality of depth pixels having respective depth values (the depth image includes depth pixels at specific points x,y and finding a value of depth at the position; fig 6 and 8; paragraphs [0208-0212]), with the depth image representing at least a portion of the object, generate a brightness image different from the depth image based on the reflected light (the computing system further includes a color image (brightness image) which comprises a brightness/color value related and at the same position as the depth pixel of the depth image but the color image is a different image seen in fig 5; figs 5-6 and 8; paragraphs [0177], [0224-0225]), the brightness image comprising a plurality of brightness pixels and representing at least the portion of the object, each brightness pixel of the plurality of brightness pixels corresponding to a respective depth pixel of the plurality of depth pixels, for each depth pixel of the plurality of depth pixels (each pixel is mapped in the images captured at a specific time T such that the pixel in the same x,y position in the depth image representation and the color image representation is the same pixel in each representation and allows the user to find a first depth value of the position related to the selected brightness pixel position; fig 6, and 8-9; paragraphs [0093], [0126-0127], [0135], [0145], [0174-0175], [0212], [0252], [0257]). CHO fails to disclose a system, comprising: an illuminator assembly configured to project modulated light into a local area including an object; determine a respective energy based on a first gradient magnitude of the respective depth pixel in the depth image and a different second gradient magnitude of a brightness pixel in the brightness image, determine enhanced depth values of respective ones of the plurality of depth pixels by fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels and generate an enhanced depth image based on the enhanced depth values by, at least in part, replacing a particular depth value of the respective depth values with a particular enhanced death value of the enhanced depth values, with both the particular depth value and the particular enhanced depth value being associated with a particular depth pixel of the plurality of depth pixels.
CHEN discloses a system, comprising: an illuminator assembly configured to project modulated light into a local area including an object (a computing system for determining depth images and depth pixels is adapted to include light source 1 to project light of a certain energy level onto a field of view of a camera including an object to be imaged; paragraphs [0037-0039]); determine a respective energy based on a first gradient magnitude of the respective depth pixel in the depth image and a different second gradient magnitude of a brightness pixel in the brightness image (during step s09 the computing system is adapted to register a first fused depth map with a first and second image comprising target pixels and generate 3D point could data for each pixel/point in each respective image/feature map generated is generated using higher accuracy, and would include an energy value; figs 2A-B; paragraphs [0032-0034], [0040-0041]), determine enhanced depth values of respective ones of the plurality of depth pixels by fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels and generate an enhanced depth image based on the enhanced depth values by, at least in part (during step s09 the computing system is adapted to register a first fused depth map with a first and second image comprising target pixels and generate 3D point could data for each pixel/point in each respective image/feature map generated is generated using higher accuracy, and would include an energy value; figs 2A-B; paragraphs [0032-0034], [0040-0041]), replacing a particular depth value of the respective depth values with a particular enhanced death value of the enhanced depth values (for example as stated in paragraph 0041 the first preset pixel value is set to 5000 the second preset pixel value is 20,000, the first preset depth value is 10 meters the second preset depth value is 0.5 meters; paragraphs [0032-0034], [0040-0041]), with both the particular depth value and the particular enhanced depth value being associated with a particular depth pixel of the plurality of depth pixels (the aforementioned depth values correspond to different image iterations of the same target pixel, assigning a new depth value to the target pixel as desired based on parameter weighting and generating a map with desired accuracy, wherein the second depth value assigned to the target pixel is 0.5 meters; paragraphs [0032-0034], [0040-0041]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have with both the particular depth value and the particular enhanced depth value being associated with a particular depth pixel of the plurality of depth pixels of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 11.
As per claim 12, CHO in view of CHEN discloses the system of claim 11. CHO fails to disclose wherein fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels comprises: for each depth pixel of the plurality of depth pixels, optimizing the respective energy.
CHEN discloses wherein fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels comprises: for each depth pixel of the plurality of depth pixels, optimizing the respective energy (the energy value is determined to be enough to power light source however included in the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source 1, so as to prevent unnecessary power consumption (optimizing energy usage over the system); paragraphs [0040-0041], [0046]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have for each depth pixel of the plurality of depth pixels, optimizing the respective energy of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 12.
As per claim 14, CHO in view of CHEN discloses the system of claim 11. Modified CHO fails to disclose wherein the controller is further configured to determine the respective energy based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image further by: determining a conditional error energy for the respective depth pixel based on a depth value of the respective depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the respective depth pixel given the brightness value of the brightness pixel.
CHEN discloses wherein the controller is further configured to determine the respective energy based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image further by: determining a conditional error energy for the respective depth pixel based on a depth value of the respective depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image (the energy value is determined to be enough to power light source however included in the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power (conditional energy) should be supplied to the light source 1, so as to prevent unnecessary power consumption (increase in error); paragraphs [0040-0041], [0046]), wherein the conditional error energy indicates a measure of uncertainty in the depth value of the respective depth pixel given the brightness value of the brightness pixel (the processor 4 is programmed to control the light source to be driven by a second driving current I2, which is not greater than a minimum of the first driving current I1 (and provided the difference in currents which is the value of uncertainty as it will depend on exact current values of the energy components used to power the light source), to emit light, the first driving current I1 and the second driving current I2 are generated by the processor; paragraphs [0040-0041], [0046]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 14.
As per claim 16, CHO discloses one or more non-transitory computer-readable storage media storing instructions executable to perform operations, the operations comprising (a computing system comprising a memory to store instructions, data, and programs to perform image processing methods related to view extrapolation; abstract; figs 1 and 3): determining a boundary weight for a target depth pixel of a plurality of depth pixels based on a first gradient magnitude of the target depth pixel in a depth image (the system is adapted to find an object boundary hole acting substantially as a boundary point and has a boundary weight related to a depth pixel of a depth image and is adapted to find the gradient depth value of the specific pixel point; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]) and a different second gradient magnitude of a brightness pixel in a brightness image different from the depth image (the computing system further includes a color image (brightness image) which comprises a brightness/color value related and at the same position as the depth pixel of the depth image but the color image is a different image seen in fig 5; figs 5-6 and 8; paragraphs [0177], [0224-0225]), with the depth image comprising the plurality of depth pixels (the depth image includes depth pixels at specific points x,y and finding a value of depth at the position; fig 6 and 8; paragraphs [0208-0212]), and with the target depth pixel having a first depth value and corresponding to the brightness pixel (each pixel is mapped in the images captured at a specific time T such that the pixel in the same x,y position in the depth image representation and the color image representation is the same pixel in each representation and allows the user to find a first depth value of the position related to the selected brightness pixel position; fig 6, and 8-9; paragraphs [0093], [0126-0127], [0135], [0145], [0174-0175], [0212], [0252], [0257]). CHO fails to disclose determining an energy for the target depth pixel based on the boundary weight; determining a second depth value of the target depth pixel by optimizing the energy; and updating the depth image by assigning the second depth value to the target depth pixel.
CHEN discloses determining an energy for the target depth pixel based on the boundary weight (during step s09 the computing system is adapted to register a first fused depth map with a first and second image comprising target pixels and generate 3D point could data for each pixel/point in each respective image/feature map generated is generated using higher accuracy, and would include an energy value; figs 2A-B; paragraphs [0032-0034], [0040-0041]); determining a second depth value of the target depth pixel by optimizing the energy (for example as stated in paragraph 0041 the first preset pixel value is set to 5000 the second preset pixel value is 20,000, the first preset depth value is 10 meters the second preset depth value is 0.5 meters; paragraphs [0032-0034], [0040-0041]); and updating the depth image by assigning the second depth value to the target depth pixel (the aforementioned depth values correspond to different image iterations of the same target pixel, assigning a new depth value to the target pixel as desired based on parameter weighting and generating a map with desired accuracy, wherein the second depth value assigned to the target pixel is 0.5 meters; paragraphs [0032-0034], [0040-0041]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have and updating the depth image by assigning the second depth value to the target depth pixel of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 16.
As per claim 17, CHO in view of CHEN discloses the one or more non-transitory computer-readable storage media of claim 16. Modified CHO further discloses wherein: the brightness image and the depth image represent a same object (one pixel may be expressed in color and in depth, the input sequence includes a color sequence representing information about colors of pixels forming a frame and a depth sequence representing information about depth of the pixels and since the pixel may be at the same point in each image the object at that point would also be the same; fig 5; paragraph [0135]), the brightness image comprises a plurality of brightness pixels that includes the brightness pixel (the color image includes a plurality of pixels each comprising their own respective brightness/color values assigned to each individual pixel at its individual point; fig 5; paragraph [0135]), and each respective brightness pixel of the plurality of brightness pixels correspond to a respective depth pixel of the plurality of depth pixels (each respective pixel of the color/brightness image at a specific time T corresponds to a specific position of a depth pixel at the same time T in a depth image; fig 5; paragraph [0135]).
As per claim 19, CHO in view of CHEN discloses the one or more non-transitory computer-readable storage media of claim 18. CHO fails to disclose wherein determining the energy for the target depth pixel based on the boundary weight further comprises: determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel.
CHEN discloses wherein determining the energy for the target depth pixel based on the boundary weight further comprises: determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image (the energy value is determined to be enough to power light source however included in the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power (conditional energy) should be supplied to the light source 1, so as to prevent unnecessary power consumption (increase in error); paragraphs [0040-0041], [0046]), wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel (the processor 4 is programmed to control the light source to be driven by a second driving current I2, which is not greater than a minimum of the first driving current I1 (and provided the difference in currents which is the value of uncertainty as it will depend on exact current values of the energy components used to power the light source), to emit light, the first driving current I1 and the second driving current I2 are generated by the processor; paragraphs [0040-0041], [0046]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 19.
As per claim 20, CHO in view of CHEN discloses the one or more non-transitory computer-readable storage media of claim 16. Modified CHO further discloses wherein the depth image and the brightness image are generated based on image data from a same image sensor (the brightness/color image and the depth image are captured by the system using a camera sensor; fig 5; paragraphs [0051], [0142]).
As per claim 21, CHO in view of CHEN discloses the method of claim 1. Modified CHO further discloses wherein determining the boundary weight for the target depth pixel comprises (the system is adapted to find an object boundary hole acting substantially as a boundary point and has a boundary weight related to a depth pixel of a depth image and is adapted to find the gradient depth value of the specific pixel point; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]); determining a product of the first gradient magnitude of the target depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image, resulting in a fusion gradient magnitude for the target depth pixel (the importance of the vertical line 870 may be a value obtained by multiplying (taking a product of) a sum of a gradient value of each vertical line and a depth value by a position weighting related to the x,y position of the pixel in the depth image and in the brightness/color image; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]); and determining, using the fusion gradient magnitude, the boundary weight, with the boundary weight representing a combination of one or more boundaries in the depth image and one or more additional boundaries in the brightness image (the system is adapted to find an object boundary hole acting substantially as a boundary point and has a boundary weight related to a depth pixel of a depth image and is adapted to find the gradient depth value of the specific pixel point in both the color/brightness image and the depth image provided in fig 5; figs 3 and 5; paragraphs [0068], [0076], [0104], [0166], [0232-0233]).
As per claim 22, CHO in view of CHEN discloses the method of claim 1. CHO fails to disclose wherein optimizing the energy comprises minimizing the energy.
CHEN discloses wherein optimizing the energy comprises minimizing the energy (the energy value is determined to be enough to power light source however included in the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source 1, so as to prevent unnecessary power consumption (optimizing/minimizing energy usage over the system); paragraphs [0040-0041], [0046]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have wherein optimizing the energy comprises minimizing the energy of CHEN reference. The Suggestion/motivation for doing so would have been to provide the ability to by virtue of determining the number of edge pixels and the depth representative value, the control method of a depth camera according to this disclosure is capable of deciding whether to generate the second depth map and, in certain cases, how much operational power should be supplied to the light source, so as to prevent unnecessary power consumption while maintaining high accuracy of depth detection of the depth camera as suggested by paragraph [0046]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine CHEN with CHO to obtain the invention as specified in claim 22.
Claims 4, 13, and 18 are rejected under 35 § U.S.C. 103 as being obvious over US 2012/0033872 A1 to CHO et al. (hereinafter “CHO”), in view of US 2016/0330434 A1 to CHEN (hereinafter “CHEN”) in further view of US 2007/0110319 A1 to WYATT et al. (hereinafter “WYATT”)
As per claim 4, CHO in view of CHEN discloses the method of claim 1. Modified CHO fails to disclose wherein determining the energy for the target depth pixel based on the boundary weight comprises: determining a spatial error energy for the target depth pixel based on the boundary weight, wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image.
WYATT discloses wherein determining the energy for the target depth pixel based on the boundary weight comprises: determining a spatial error energy for the target depth pixel based on the boundary weight (the first and second gradient magnitudes are found in the x and y direction of the target pixel and is used to determine the spatial derivative value of each gradient magnitude and in a direction parallel to the edge direction, it can be assumed that brightness gradient values originating from edges are not included in the image and that only spatial derivative values originating from noise (spatial error) are included in the image; paragraphs [0004], [0008], [0032], [0038-0041], [0082]), wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image (the depth pixels having the brightness gradient values are selected in the direction which the gradient value maximizes (optimize) by as seen in equation (2) taking a difference of gradient values of the selected target/depth pixel and is adjacent to the gradient/related pixel being used in the equation and this is done to reduce noise/error in the in the image; figs 1-2; paragraphs [0074-0081]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a spatial error energy for the respective depth pixel based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image of WYATT reference. The Suggestion/motivation for doing so would have been to provide the ability to determine edge orientation and direction based on the way the brightness gradient maximizes and minimizes as suggested by WYATT at paragraph [0044]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine WYATT with CHO to obtain the invention as specified in claim 4.
As per claim 13, CHO in view of CHEN discloses the system of claim 12. CHO fails to disclose wherein the controller is further configured to determine the respective energy based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image by: determining a spatial error energy for the respective depth pixel based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image, wherein optimizing the respective energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the respective depth pixel and a depth value of another depth pixel that is adjacent to the respective depth pixel in the depth image.
WYATT discloses wherein the controller is further configured to determine the respective energy based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image by: determining a spatial error energy for the respective depth pixel based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image (the first and second gradient magnitudes are found in the x and y direction of the target pixel and is used to determine the spatial derivative value of each gradient magnitude and in a direction parallel to the edge direction, it can be assumed that brightness gradient values originating from edges are not included in the image and that only spatial derivative values originating from noise (spatial error) are included in the image; paragraphs [0004], [0008], [0032], [0038-0041], [0082]), wherein optimizing the respective energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the respective depth pixel and a depth value of another depth pixel that is adjacent to the respective depth pixel in the depth image (the depth pixels having the brightness gradient values are selected in the direction which the gradient value maximizes (optimize) by as seen in equation (2) taking a difference of gradient values of the selected target/depth pixel and is adjacent to the gradient/related pixel being used in the equation and this is done to reduce noise/error in the in the image; figs 1-2; paragraphs [0074-0081]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a spatial error energy for the respective depth pixel based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image of WYATT reference. The Suggestion/motivation for doing so would have been to provide the ability to determine edge orientation and direction based on the way the brightness gradient maximizes and minimizes as suggested by WYATT at paragraph [0044]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine WYATT with CHO to obtain the invention as specified in claim 13.
As per claim 18, CHO in view of CHEN discloses the one or more non-transitory computer-readable storage media of claim 16. CHO fails to disclose wherein determining the energy for the target depth pixel based on the boundary weight comprises: determining a spatial error energy for the target depth pixel based on the boundary weight, wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image.
WYATT discloses wherein determining the energy for the target depth pixel based on the boundary weight comprises: determining a spatial error energy for the target depth pixel based on the boundary weight (the first and second gradient magnitudes are found in the x and y direction of the target pixel and is used to determine the spatial derivative value of each gradient magnitude and in a direction parallel to the edge direction, it can be assumed that brightness gradient values originating from edges are not included in the image and that only spatial derivative values originating from noise (spatial error) are included in the image; paragraphs [0004], [0008], [0032], [0038-0041], [0082]), wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image (the depth pixels having the brightness gradient values are selected in the direction which the gradient value maximizes (optimize) by as seen in equation (2) taking a difference of gradient values of the selected target/depth pixel and is adjacent to the gradient/related pixel being used in the equation and this is done to reduce noise/error in the in the image; figs 1-2; paragraphs [0074-0081]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify CHO to have determining a spatial error energy for the respective depth pixel based on the first gradient magnitude of the respective depth pixel in the depth image and the second gradient magnitude of the brightness pixel in the brightness image of WYATT reference. The Suggestion/motivation for doing so would have been to provide the ability to determine edge orientation and direction based on the way the brightness gradient maximizes and minimizes as suggested by WYATT at paragraph [0044]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine WYATT with CHO to obtain the invention as specified in claim 18.
Claims 8-9 and 15 are rejected under 35 § U.S.C. 103 as being obvious over US 2012/0033872 A1 to CHO et al. (hereinafter “CHO”), in view of US 2016/0330434 A1 to CHEN (hereinafter “CHEN”) in further view of US 2021/0356598 A1 to HURWITZ (hereinafter “HURWITZ”).
As per claim 8, CHO in view of CHEN discloses the method of claim 1. Modified CHO fails to disclose further comprising: instructing an illuminator assembly to project modulated light onto a local area including an object; instructing a camera assembly to capture reflected light from at least a portion of the object; and generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area.
HURWITZ discloses further comprising: instructing an illuminator assembly to project modulated light into a local area including an object (a light from any light source is projected towards an object 910 and the light is reflected off of said object; fig 10a; paragraphs [0097], [0122]); instructing a camera assembly to capture reflected light from at least a portion of the object (instructing a camera 500 to capture the reflected light from the object 910; fig 10a; paragraphs [0097], [0122]); and generating the depth image based on a phase shift between the reflected light and the modulated light projected onto the local area (the system including controller 140 for determining a depth image/frame using determined phase relationship between the first laser light and the received reflected light and the determined phase relationship between the second laser light and the received reflected light, phase unwrapping may be performed to arrive at said depth image frame; paragraphs [0070], [0164]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to further modify CHO to have generating the depth image based on a phase shift between the reflected light and the modulated light projected onto the local area and reflected off the object of HURWITZ reference. The Suggestion/motivation for doing so would have been to use correlated double sampling which is performed to minimize KTC noise as suggested by HURWITZ at paragraphs [0073]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine HURWITZ with modified CHO to obtain the invention as specified in claim 8.
As per claim 9, CHO in view of CHEN in view of HURWITZ discloses the method of claim 8. Modified CHO fails to disclose further comprising: generating the brightness image based on brightness of the reflected light.
HURWITZ discloses further comprising: generating the brightness image based on brightness of the reflected light (a 2D IR frame may also be determined using the determined active brightness for the first laser light and/or the determined active brightness for the second laser light and with the image acquisition components 130, 140 and 150 reconfigured to control pulsed emission from the laser 110 and determine a depth frame based on a time difference between emission of a pulse and reception of reflected light/brightness image a 2D IR frame may also be determined based on the magnitude of charge accumulated in the imaging pixels of the image sensor 120; paragraph [0069-0072]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to further modify CHO to have generating the brightness image based on brightness of the reflected light of HURWITZ reference. The Suggestion/motivation for doing so would have been to use correlated double sampling which is performed to minimize KTC noise as suggested by HURWITZ at paragraphs [0073]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine HURWITZ with modified CHO to obtain the invention as specified in claim 9.
As per claim 15, CHO in view of CHEN discloses the system of claim 11. Modified CHO fails to disclose wherein to generate the depth image and the brightness image based on the reflected light by: generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and generating the brightness image based on brightness of the reflected light.
HURWITZ discloses wherein to generate the depth image and the brightness image based on the reflected light the controller is configured to: generate the depth image based on a phase shift between the reflected light and the modulated light projected into the local area (imaging pixels of the image sensor are given phase offsets according to the phase off set equation table provided in paragraph [0065] where the skilled person will readily understand that using DFT to determine the phase relationship between the first laser light and the received reflected laser light, and to determine active brightness, is merely one example and that any other suitable alternative technique may be used; paragraphs [0056], [0063-0065]); and generate the brightness image based on brightness of the reflected light (determine an active brightness 2D IR image frame based on the reflected light; paragraphs [0056-0057], [0068-0069]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to further modify CHO to have generated depth maps based on reflected light of the object of HURWITZ reference. The Suggestion/motivation for doing so would have been to provide as suggested by HURWITZ at paragraph [0070] that this process may be repeated many times in order to generate a time series of depth frames, which may together form a video of the depth image frames in sequence. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine HURWITZ with modified CHO to obtain the invention as specified in claim 15.
Claim 10 is rejected under 35 § U.S.C. 103 as being obvious over US 2012/0033872 A1 to CHO et al. (hereinafter “CHO”), in view of US 2016/0330434 A1 to CHEN (hereinafter “CHEN”) in further view of US 2021/0356598 A1 to HURWITZ (hereinafter “HURWITZ”) in further view of US 2017/0018114 A1 to STEWART et al (hereinafter “STEWART”).
As per claim 10, CHO in view of CHEN in further view of HURWITZ discloses the method of claim 8. Modified CHO fails to disclose wherein the reflected light is first reflected light, and the method further comprises: instructing the camera assembly to capture second reflected light from at least the portion of the object; and generating the brightness image based on brightness of the second reflected light, wherein the second reflected light has a different wavelength from the first reflected light.
STEWART discloses wherein the reflected light is first reflected light, and the method further comprises: instructing the camera assembly to capture second reflected light from at least the portion of the object (the camera and corresponding probe light from modulated emitter 32 is to apply probe light to subject 16’ at areas 34 and second area 36 and to determine reflectance and corresponding wavelength of the two reflected areas off of subject 16’; fig 1; paragraphs [0017-0018], [0023-0024]); and generating the brightness image based on brightness of the second reflected light, wherein the second reflected light has a different wavelength from the first reflected light (sensor 18 adapted to capture the reflectance of light off of the subject and includes one or more passive filters 22 may be arranged in series with sensor array 14 and configured to limit the wavelength response of the sensor array passive filters reduce noise by excluding photons of wavelengths not intended to be imaged and then a map of the reflected brightness’s from areas 34 and 36 may be generated based on the desired wavelengths allowed through from the filter arrangement; fig 1; paragraphs [0017-0018], [0023-0024], [0038]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to further modify CHO to have wherein the second reflected light has a different wavelength from the first reflected light of STEWART reference. The Suggestion/motivation for doing so would have been to filter out wavelengths of light reflectance not desired for observation as suggested by STEWART at paragraphs [0018-0019]. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine STEWART with modified CHO to obtain the invention as specified in claim 10.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Examiner's Note: Examiner has cited figures, and paragraphs in the references as applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested for the applicant, in preparing the responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Examiner has also cited references in PTO892 but not relied on, which are relevant and pertinent to the applicant’s disclosure, and may also be reading (anticipatory/obvious) on the claims and claimed limitations. Applicant is advised to consider the references in preparing the response/amendments in-order to expedite the prosecution.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVIN JACOB DHOOGE whose telephone number is (571) 270-0999. The examiner can normally be reached 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800- 786-9199 (IN USA OR CANADA) or 571-272-1000.
/Devin Dhooge/
USPTO Patent Examiner
Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights