DETAILED ACTION
Notice of Pre-AIA or AIA Status
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 04/20/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
AMENDMENTS
Applicant/s submitted arguments and remarks on 04/20/2026. The Examiner acknowledges the arguments and reviewed the claims accordingly.
Applicant/s amended claims 1, 8, and 12 – 15. Claims 1 – 20 are currently pending.
Response to Arguments
In regards to Argument 1, with respect to the rejection of claims 1 – 20 under 35 U.S.C. 101 for being directed to an abstract idea of mental process, the applicant/s states that the independent claims 1 and 15 have been amended. The applicant/s further states that the currently-amended claim limitations recite features that are not mental processes and cannot be performed using a generic computer program. The applicant/s states that such recitations require specialized computing instructions capable of causing a depth mode to generate depth information based on sensor data at a specified rate and specialized instructions capable of depth-based image processing. Therefore, the applicant/s request the withdrawal of rejection of claims 1 – 20 under 35 U.S.C. 101. (See Remarks page 8 – 9, dated, 04/20/2026)
In response to Argument 1, with respect to the rejection of claims 1 – 20 under 35 U.S.C. 101 for being directed to an abstract idea of mental process, the Examiner states that the applicant/s arguments have been fully considered but are rendered moot in view of amendments made to the independent claims. Therefore, the rejection of claims 1 – 20 under 35 U.S.C. 101 have been withdrawn. However, upon further consideration, the claims have been found to be eligible under 35 U.S.C 101.
In regards to Argument 2, with respect to the rejection of claims 1, 6 – 9, 12, 15 and 20 under 35 U.S.C. 102, the applicant/s states that the independent claims 1 and 15 have been amended. The applicant/s further states that Grossinger does not teach the limitations recited in the amended independent claims. Therefore, the applicant/s requests the withdrawal of rejection of amended independent claims 1 and 15 and their dependent claims respectively under 35 U.S.C. 102. (See Arguments/Remarks, page 9 – 10, dated 04/20/2026)
In response to Argument 2, with respect to the rejection of claims 1, 6 – 9, 12, 15 and 20 under 35 U.S.C. 102, the Examiner states that the applicant/s arguments have been fully considered but are rendered moot in view of the amendments made to the independent claims. Therefore, the Examiner states that the rejection of claims under 35 U.S.C. 102 have been withdrawn. However, upon further search and consideration, the following new grounds of rejections have been necessitated by the amendments.
In regards to Argument 3, with respect to the rejection of claims 2 – 5, 10, 11, 13 – 14, and 16 – 19 under 35 U.S.C. 103, the applicant/s states that Grossinger fails to teach the currently amended independent claims 1 and 15 and Zobel and Wantland fails to make up for the deficiencies of Grossinger. Therefore, the applicant/s requests the withdrawal of rejection of claims under 35 U.S.C. 103. (See Arguments/Remarks, page 11, dated 04/20/2026)
In response to Argument 3, with respect to the rejection of claims 2 – 5, 10, 11, 13 – 14, and 16 – 19 under 35 U.S.C. 103, the Examiner states that the applicant/s arguments have been fully considered but are rendered moot in view of the amendments made to the independent claims. However, upon further search and consideration the following new grounds of rejections have been necessitated by the amendments made to the independent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 6 – 9, 12, 14 – 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger) in view of Smith et al. (US 20230396881 A1; hereafter referred to as Smith).
Regarding Claim 1, Grossinger teaches:
An apparatus for processing images, the apparatus comprising:
at least one memory (Grossinger, [0112] “Such a computer program may be stored in a non-transitory, tangible computer readable storage medium”); and
at least one processor coupled to the at least one memory (Grossinger, [0100] “. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user”) and configured to:
obtain scene information based on a scene (Grossinger, [0022] “The plurality of cameras captures a set of images of the local area. The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”; Grossinger, [0033] “determine depth of a scene”);
determine a depth scheme from among a plurality of depth schemes based on the scene information (Grossinger, [0033] The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”);
use the depth scheme to obtain the depth information of the scene (Grossinger, [0034] The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”); and
process an image of the scene based on the depth information (Grossinger, [0034] “creates or updates a depth model describing the local area based on the depth information”; Grossinger, [0042] “The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area (e.g., the depth model), update a model of the local area, or some combination thereof”).
However, Grossinger does not explicitly recite:
wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode;
In the same field of endeavor, Smith teaches:
wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode (Smith, [0046] Each application 310 can make requests to the depth sensing system 300 for various types of depth information as needed. The arbiter 330 is responsible for receiving the requests for depth information and for scheduling the depth sensing operations which will provide the requested depth information… the arbiter 330 prioritizes depth sensing requests in the following order (though other prioritization schemes can also be used): 1) short-range, high frame rate depth measurements; 2) high dynamic range depth measurements (made up of short-range, low frame rate depth measurements interleaved with long-range, low frame rate depth measurements); 3) short-range, low frame rate depth measurements; 4) long-range, high frame rate depth measurements; 5) long-range, low frame rate depth measurements; and 6) idle state”; Smith, [0050] “the first depth sensing mode may be a short-range, high frame rate mode… the second depth sensing mode may be a long-range, high frame rate mode…”; Smith, Fig. 1, [0051] – [0052]; Smith, [0061] “FIG. 6 illustrates another example of an improved method 600 for efficiently operating the depth sensor 100 in multiple depth sensing modes”);
Grossinger and Smith are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Smith to make the invention wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode; doing so can result in computer-generated imagery that create the impression of being three-dimensional and present stereoscopic imagery to the user (Smith [0017]); thus, one of the ordinary skill on the art would have been motivated to combine the references.
Regarding Claim 6, Grossinger in view of Smith teaches the apparatus of claim 1, wherein the scene information comprises at least one of:
information based on an object in the scene (Grossinger, [0034] “depth sensing condition may be, e.g., an environmental condition of the local area (e.g., an ambient light level), a location of objects in the local area (e.g., the distance to an object from the DCA)”);
information indicative of a confidence of the depth information (Grossinger, [0024] “the DCA may determine a confidence map associated with the depth model and may input the depth model, the confidence map, and the set of captured images into the machine learning model. The machine learning model outputs the refined depth model”);
information indicative of motion of a device that obtains the depth information (Grossinger, [0095] “The position sensor 840 is an electronic device that generates data indicating a position of the headset 805. The position sensor 840 generates one or more measurement signals in response to motion of the headset 805”);
information indicative of motion within the scene (Smith, [0029] “Once the fact that the user's hand is present in the field of view of the depth sensor has been detected, close range, high frame rate depth measurements may be more useful for tracking the movements of the user's hands and thereby detecting a specific gesture being made”);
information indicative of lighting within the scene (Grossinger, [0034] “the illuminator 140 illuminates a portion of the local area with light. The light may be, e.g., structured light (e.g., dot pattern, bars, etc.) in the infrared (IR), uniform light illuminating a scene, IR flash for time-of-flight, etc. In some embodiments, the one or more imaging devices 130 capture a set of images of the portion of the local area that include the light from the illuminator 140”);
information related to of the depth information (Grossinger, [0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”); or
tone/color/tint information related to the scene (Grossinger, [0082] “The example depth model 617 is depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area. The example refined depth model 642 is an example of the refined depth model 640 determined by the machine learning model 630. The example refined depth model 642 is also depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area”).
Regarding Claim 7, Grossinger in view of Smith teaches the apparatus of claim 6, wherein the information based on the object comprises at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
information related to a classification of the object;
information indicative of motion of the object (Smith, [0029] “Once the fact that the user's hand is present in the field of view of the depth sensor has been detected, close range, high frame rate depth measurements may be more useful for tracking the movements of the user's hands and thereby detecting a specific gesture being made”);
information indicative of a depth of the object in the scene (Grossinger, [0053] “Depth measurements may include one or more distances between a device the DCA is a component of and one or more real-world objects in the local area, one or more distances between two or more real-world objects in the local area, an orientation of the user of the device the DCA is a component of in the local area, and so on”; Grossinger, [0068] “calculating a depth to an object”); or
information indicative of a confidence of the information indicative of the depth of the object in the scene (Grossinger, [0060] “The depth determination module 250 determines the associated confidence map by determining a confidence value for each pixel of the corresponding depth model. For example, the depth determination module 250 compares the depth model to the set of captured images and assigns a confidence value to each pixel of the depth model based on the comparison”).
Regarding Claim 8, Grossinger in view of Smith teaches the apparatus of claim 1, wherein the depth mode is selected from plurality of depth modes (Grossinger, [0033] “The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”; Grossinger, [0034] “The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”), the one or more depth modes comprising at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique (Grossinger, [0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique (Grossinger, [0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity”; Grossinger, [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 9, Grossinger in view of Smith teaches the apparatus of claim 1, wherein, to determine the depth scheme, the at least one processor is configured to determine to obtain the depth information based on two or more depth modes (Grossinger, [0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”).
Regarding Claim 12, Grossinger in view of Smith teaches the apparatus of claim 1, wherein the at least one processor is configured to adjust, based on the scene information, a rate at which the depth scheme determines the depth information (Grossinger, [0023] “The DCA may switch between depth sensing modes when preferable. The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; Smith, [0029] “close range, low frame rate depth measurements may be sufficient for detecting when the user's hand is present in the field of view of the depth sensor 100. Once the fact that the user's hand is present in the field of view of the depth sensor has been detected, close range, high frame rate depth measurements may be more useful for tracking the movements of the user's hands and thereby detecting a specific gesture being made. Meanwhile, long-range depth measurements at low or high frame rates can be useful for mapping the user's environment”; Smith, [0077] “The method 600 of operation shown in FIG. 6 … can improve depth sensing efficiency by reducing the amount of time which is dedicated to programming the depth sensor 100 in response to changes in the requested depth sensing mode of operation”).
Regarding Claim 14, Grossinger in view of Smith teaches the apparatus of claim 1, wherein the rate at which the depth scheme determines the depth information (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes) is determined separately from an image-capture rate (Smith, Table 1; Smith, [0036] “the short-range, low frame rate mode of operation also includes a relatively long delay as step 5 of the operation sequence. This delay can be equal to, for example, the difference between the 125 ms period of the operation sequence and the total time required to complete steps 0-4. The relatively long delay of step 5 occupies the time during the period of the operation sequence which is not required in order to capture and read out the intensity sub-frame and the four phase sub-frames”; Smith, [0038] The example sequence of operations for the long-range, high frame rate depth sensing mode begins with step 0, which is obtaining the intensity sub-frame. Then, during steps 1-4, the four phase sub-frames for the first modulation frequency, Fmod1, are captured, while, during steps 5-8, the four sub-frames for the second modulation frequency, Fmod2, are captured. For long-range measurements, the exposure time (i.e., the time during which the image sensor captures light) for each of these sub-frames is longer than for short-range measurements, typically 2-3 ms. (Other parameters or settings for long-range sub-frames may also differ from short-range sub-frames”).
Regarding Claim 15, Grossinger teaches:
A method for processing images, the method comprising:
obtaining scene information based on a scene ([0022] “The plurality of cameras captures a set of images of the local area. The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”; [0033] “determine depth of a scene”);
determining a depth scheme from among a plurality of depth schemes based on the scene information ([0033] The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”);
using the depth scheme to obtain depth information of the scene ([0034] The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”); and
processing an image of the scene based on the depth information ([0034] “creates or updates a depth model describing the local area based on the depth information”; [0042] “The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area (e.g., the depth model), update a model of the local area, or some combination thereof”).
However, Grossinger does not explicitly recite:
wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode;
In the same field of endeavor, Smith teaches:
wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode (Smith, [0046] Each application 310 can make requests to the depth sensing system 300 for various types of depth information as needed. The arbiter 330 is responsible for receiving the requests for depth information and for scheduling the depth sensing operations which will provide the requested depth information… the arbiter 330 prioritizes depth sensing requests in the following order (though other prioritization schemes can also be used): 1) short-range, high frame rate depth measurements; 2) high dynamic range depth measurements (made up of short-range, low frame rate depth measurements interleaved with long-range, low frame rate depth measurements); 3) short-range, low frame rate depth measurements; 4) long-range, high frame rate depth measurements; 5) long-range, low frame rate depth measurements; and 6) idle state”; Smith, [0050] “the first depth sensing mode may be a short-range, high frame rate mode… the second depth sensing mode may be a long-range, high frame rate mode…”; Smith, Fig. 1, [0051] – [0052]; Smith, [0061] “FIG. 6 illustrates another example of an improved method 600 for efficiently operating the depth sensor 100 in multiple depth sensing modes”);
Grossinger and Smith are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Smith to make the invention wherein the depth scheme comprises a depth mode and a rate for determining depth information using the depth mode; doing so can result in computer-generated imagery that create the impression of being three-dimensional and present stereoscopic imagery to the user (Smith [0017]); thus, one of the ordinary skill on the art would have been motivated to combine the references.
Regarding Claim 20, Grossinger in view of Smith teaches the method of claim 15, wherein the scene information comprises at least one of:
information based on an object in the scene (Grossinger, [0034] “depth sensing condition may be, e.g., an environmental condition of the local area (e.g., an ambient light level), a location of objects in the local area (e.g., the distance to an object from the DCA)”);
information indicative of a confidence of the depth information (Grossinger, [0024] “he DCA may determine a confidence map associated with the depth model and may input the depth model, the confidence map, and the set of captured images into the machine learning model. The machine learning model outputs the refined depth model”);
information indicative of motion of a device that obtains the depth information (Grossinger, [0095] “The position sensor 840 is an electronic device that generates data indicating a position of the headset 805. The position sensor 840 generates one or more measurement signals in response to motion of the headset 805”);
information indicative of motion within the scene (Smith, [0029] “Once the fact that the user's hand is present in the field of view of the depth sensor has been detected, close range, high frame rate depth measurements may be more useful for tracking the movements of the user's hands and thereby detecting a specific gesture being made”);
information indicative of lighting within the scene (Grossinger, [0034] “the illuminator 140 illuminates a portion of the local area with light. The light may be, e.g., structured light (e.g., dot pattern, bars, etc.) in the infrared (IR), uniform light illuminating a scene, IR flash for time-of-flight, etc. In some embodiments, the one or more imaging devices 130 capture a set of images of the portion of the local area that include the light from the illuminator 140”);
information related to of the depth information (Grossinger, [0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”); or
tone/color/tint information related to the scene (Grossinger, [0082] “The example depth model 617 is depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area. The example refined depth model 642 is an example of the refined depth model 640 determined by the machine learning model 630. The example refined depth model 642 is also depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area”).
Claims 2 – 5, 13 and 16 - 19 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger) in view of Smith et al. (US 20230396881 A1; hereafter referred to as Smith) further in view of Zobel et al. (US 20230216999 A1; hereafter referred to as Zobel).
Regarding Claim 2, Grossinger in view of Smith teaches the apparatus of claim 1, wherein the scene information comprises first scene information based on the scene at a first time, the depth scheme comprises a first depth scheme, the depth information comprises first depth information obtained by the first depth scheme at a second time, the image comprises a first image of the scene (Grossinger, [0055] “The depth determination module 250 may select a depth sensing mode based on one or more depth sensing conditions for regions of the local area. For example, the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes. For example, the controller 230 may obtain a first set of depth information using a structured light mode using a first camera and the illuminator”), and the at least one processor is configured to:
obtain second scene information based on the scene (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”);
determine a second depth scheme from among the plurality of depth schemes based on the second scene information, wherein the second depth scheme is different than the first depth scheme (Grossinger, [0034] “A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0067] The DCA 300 may use different depth sensing modes for different portions of the local area 310. For example, the first region 340 may be located in a shadow, and the second region 360 may be located in a well-lit area. In another example, the first region 340 may comprise a smooth wall with minimal texture, and the second region 360 may comprise multiple objects located at different depths relative to the DCA 300”);
use the second depth scheme to obtain second depth information of the scene at a fourth time (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”; Grossinger, [0055] “the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”); and
process a second image of the scene based on the second depth information (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator. The controller 230 may obtain a third set of depth information using a stereo depth sensing mode. In response to detecting a difference between expected measurements, the calibration module 260 may determine that the cameras should be calibrated, and the calibration module 260 may adjust a stereo depth matching algorithm to account for the discrepancy”).
While Grossinger in view of Smith teaches depth sensing conditions based on time of the day measurements which is broadly considered as different times, it fails to explicitly teach:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time,
obtain second scene information based on the scene at a third time;
use the second depth scheme to obtain second depth information of the scene at a fourth time;
In the same field of endeavor, Zobel teaches:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time (Zobel, [0232] “the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time….the first image data includes at least one image depicting the environment at least at one of the first time”),
obtain second scene information based on the scene at a third time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
use the second depth scheme to obtain second depth information of the scene at a fourth time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
Grossinger, Smith and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger in view of Smith with the invention of Zobel to make the invention that obtains first depth information at a second time; obtains second scene information based on the scene at a third time and obtains second depth information of the scene at a fourth time; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 3, Grossinger in view of Smith further in view of Zobel teaches the apparatus of claim 2, wherein:
to use the first depth scheme of the plurality of depth schemes to obtain the depth information, the at least one processor is configured to obtain the depth information using a first number of depth modes of a plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”); and
to use the second depth scheme of the plurality of depth schemes to obtain depth information, the at least one processor is configured to obtain the depth information using a second number of depth modes of the plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”).
Regarding Claim 4, Grossinger in view of Smith further in view of Zobel teaches the apparatus of claim 3, wherein the plurality of depth modes comprises at least two of (Examiners Note: since the claim recites “at least two of”, the Examiner is considering that at least two of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique (Grossinger, [0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique (Grossinger, [0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity” [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 5, Grossinger in view of Smith further in view of Zobel teaches the apparatus of claim 3, wherein the first number of depth modes is different from the second number of depth modes (Grossinger, [0023] “The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes).
Regarding Claim 13, Grossinger in view of Smith teaches the apparatus of claim 1, but fails to explicitly teach:
wherein the at least one processor is configured to interpolate between instances of depth information to generate interpolated depth information.
In the same field of endeavor, Zobel teaches:
wherein the at least one processor is configured to interpolate between instances of depth information to generate interpolated depth information (Zobel, [0079] “ The change in perspective can be used for frame interpolation to increase effective frame rate of a video by generating an intermediate frame in between two existing frames”; Zobel, [0134] “the smaller motion vector maps generated by the time warp 705 can be used to interpolate additional frames in between existing frames of a video, for instance to increase the frame rate of the video from a first frame rate to a second frame rate that is higher than the first frame rate”).
Grossinger, Smith and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger in view of Smith with the invention of Zobel to make the invention that interpolates between instances of depth information to generate interpolated depth information; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 16, Grossinger in view of Smith teaches the method of claim 15, wherein the scene information comprises first scene information based on the scene at a first time, the depth scheme comprises a first depth scheme, the depth information comprises first depth information obtained by the first depth scheme at a second time, the image comprises a first image of the scene (Grossinger, [0055] “The depth determination module 250 may select a depth sensing mode based on one or more depth sensing conditions for regions of the local area. For example, the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes. For example, the controller 230 may obtain a first set of depth information using a structured light mode using a first camera and the illuminator”), and the at least one processor is configured to:
obtaining second scene information based on the scene (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”);
determining a second depth scheme from among the plurality of depth schemes based on the second scene information, wherein the second depth scheme is different than the first depth scheme (Grossinger, [0034] “A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0067] The DCA 300 may use different depth sensing modes for different portions of the local area 310. For example, the first region 340 may be located in a shadow, and the second region 360 may be located in a well-lit area. In another example, the first region 340 may comprise a smooth wall with minimal texture, and the second region 360 may comprise multiple objects located at different depths relative to the DCA 300”);
using the second depth scheme to obtain second depth information of the scene (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”; Grossinger, [0055] “the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”); and
processing a second image of the scene based on the second depth information (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator. The controller 230 may obtain a third set of depth information using a stereo depth sensing mode. In response to detecting a difference between expected measurements, the calibration module 260 may determine that the cameras should be calibrated, and the calibration module 260 may adjust a stereo depth matching algorithm to account for the discrepancy”).
While Grossinger in view of Smith teaches depth sensing conditions based on time of the day measurements which is broadly considered as different times, it fails to explicitly teach:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time,
obtaining second scene information based on the scene at a third time;
using the second depth scheme to obtain second depth information of the scene at a fourth time;
In the same field of endeavor, Zobel teaches:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time (Zobel, [0232] “the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time….the first image data includes at least one image depicting the environment at least at one of the first time”),
obtaining second scene information based on the scene at a third time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
using the second depth scheme to obtain second depth information of the scene at a fourth time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
Grossinger, Smith and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger in view of Smith with the invention of Zobel to make the invention that obtains first depth information at a second time; obtains second scene information based on the scene at a third time and obtains second depth information of the scene at a fourth time; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 17, Grossinger in view of Smith further in view of Zobel teaches the method of claim 16, wherein:
using the first depth scheme of the plurality of depth schemes to obtain the depth information, the at least one processor is configured to obtain the depth information using a first number of depth modes of a plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”); and
using the second depth scheme of the plurality of depth schemes to obtain depth information, the at least one processor is configured to obtain the depth information using a second number of depth modes of the plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”).
Regarding Claim 18, Grossinger in view of Smith further in view of Zobel teaches the method of claim 17 wherein the plurality of depth modes comprises at least two of (Examiners Note: since the claim recites “at least two of”, the Examiner is considering that at least two of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique (Grossinger, [0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique (Grossinger, [0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity” [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 19, Grossinger in view of Smith further in view of Zobel teaches the method of claim 17, wherein the first number of depth modes is different from the second number of depth modes (Grossinger, [0023] “The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes).
Claims 10 – 11 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger) in view of Smith et al. (US 20230396881 A1; hereafter referred to as Smith) further in view of Wantland et al. (US 20210042950 A1; hereafter referred to as Wantland).
Regarding Claim 10, Grossinger in view of Smith teaches the apparatus of claim 1, but fails to explicitly teach:
wherein the at least one processor is configured to modify the image of the scene based on the depth information.
In the same field of endeavor, Wantland teaches:
wherein the at least one processor is configured to modify the image of the scene based on the depth information (Wantland, [0079] “By utilizing a segmentation mask or masks in combination with depth information for the same image or images, an example graphic-object addition process may allow for augmented-reality style photo editing, where virtual objects are generated and/or modified so as to more realistically interact with the real-world objects in the image or images”).
Grossinger, Smith and Wantland are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger in view of Smith with the invention of Wantland to make the invention that modifies the image of the scene based on the depth information; doing so can provide depth information of edited version of the image wherein corrections are made to remove artifacts (Wantland, [0002]- [0003]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 11, Grossinger in view of Smith further in view of Wantland teaches the apparatus of claim 10, wherein the at least one processor is configured to:
identify foreground pixels of the image based on the depth information, wherein the foreground pixels represent a foreground of the scene (Wantland, [0019] “an example mask may involve setting all pixels that correspond to an object in the foreground of an image to white and all pixels that correspond to background features or objects to black”; Wantland, [0025] “the depth map can include a depth value for each pixel in an image”); and
identify background pixels of the image based on the depth information, wherein the background pixels represent a background of the scene (Wantland, [0034] To capture dual pixels, the camera can use a sensor that captures two slightly different views of a scene. In comparing these two views, a foreground object can appear to be stationary while background objects move vertically in an effect referred to as parallax. For example, a “selfie” or image of a person taken by that person typically has the face of that person as a foreground object and may have other objects in the background”; Wantland, [0068] “A depth map may also be utilized to determine first depth information for the at least one subject, and second depth information for the at least one background area. The perspective adjustment process may then compare the first and second depth information to determine an amount of movement for a background area in the image frame, per unit of movement of at least one subject in the image frame”);
wherein the image is modified based on the foreground pixels and the background pixels (Wantland, [0096] “segmentation masks may be used to separate the background of an image from objects in the foreground of the image. Depth information for the background may then be utilized to determine an amount of blurring to be applied to the background; Wantland, [0098] “segmentation data for image 801 provides segmentation masks for at least foreground objects 806 (e.g., the table, lamp, picture, and plant), such that the background 804 can be separated from the foreground objects 806. Further, once the background 804 is separated from the foreground objects 806, a depth map for image 801 can be used to determine depth information specifically for the background 804”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights