DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematical concepts and calculations). The claim(s) recite(s) a method, and computer-readable storage medium configured to detect a focus of attention. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
• STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
• STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
o STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
o STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
o STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claims 1 and 10 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the four statutory categories? YES. Claims 1, and 15 are directed to an apparatus and a method.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims recite steps that fall into the abstract idea category of mental processes.
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
• Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
• Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
• Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
The apparatus in claim 1, and method in claim 15 comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding Claims 1 and 13: A computing device, comprising:
determine a depth scheme from among a plurality of depth schemes based on the scene information (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; recognizing parking space based on the data…);
use the depth scheme to obtain depth information of the scene (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; use the depth scheme…);
process an image of the scene based on the depth information (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; process an image…); and
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). As such, a person could mentally analyze an image and determine a fill level, either mentally or using a pen and paper. The mere nominal recitation that the various steps are being executed by a device/in a device (e.g. processing unit) does not take the limitations out of the mental process grouping.
The use of algorithm or machine learning model to identify segmented regions of pixels and then determining and performing action based on the outcome is a common pattern of data input, analysis, and output, which courts have consistently held as abstract.
The claimed functions –recognition, inspection and dividing – could be performed conceptually by a human using pen and paper, and thus fall under abstract mental steps.
Conclusions: Thus, the claims are directed to an abstract idea.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claim 1 recites additional elements:
obtain scene information based on a scene (insignificant extra-solution activity of data acquisition);
Claims 1 and 15 does/do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception.
• There is no indication that the method improves the functioning of a computer, the machine learning model, or classification itself.
Conclusion: Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? NO, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Claims 1 and 15 does/do not recite any additional elements that are not well-understood, routine or conventional.
• The claims lack an inventive concept sufficient to transform the abstract idea into patent-eligible subject matter.
• The use of model, performing recognition and inspection based on received data, is routine and conventional in the field of machine learning.
• The claims are functionally generic with no details about architecture, training, dataset specifics, or a novel arrangement of components.
Conclusion: The claims does not add significantly more than the abstract idea.
Final Determination: INELIGIBLE under 35 U.S.C. 101. The Claims 1 and 15 are: directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept.
Regarding Claims 2 – 14 and 16 – 20: the additional elements recited in the claims do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations merely recite that the functions are performed “by model” does not demonstrate a technological improvement. The additional limitations further recite calculations that are mathematical concepts and fall under abstract ideas. The claims are functionally generic with no details about architecture, training, dataset specifics, or a novel arrangement of components. Since the claims are directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept, they are ineligible under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
(g)(1) during the course of an interference conducted under section 135 or section 291, another inventor involved therein establishes, to the extent permitted in section 104, that before such person’s invention thereof the invention was made by such other inventor and not abandoned, suppressed, or concealed, or (2) before such person’s invention thereof, the invention was made in this country by another inventor who had not abandoned, suppressed, or concealed it. In determining priority of invention under this subsection, there shall be considered not only the respective dates of conception and reduction to practice of the invention, but also the reasonable diligence of one who was first to conceive and last to reduce to practice, from a time prior to conception by the other.
A rejection on this statutory basis (35 U.S.C. 102(g) as in force on March 15, 2013) is appropriate in an application or patent that is examined under the first to file provisions of the AIA if it also contains or contained at any time (1) a claim to an invention having an effective filing date as defined in 35 U.S.C. 100(i) that is before March 16, 2013 or (2) a specific reference under 35 U.S.C. 120, 121, or 365(c) to any patent or application that contains or contained at any time such a claim.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 6 – 9, 12, 15 and 20 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger).
Regarding Claim 1, Grossinger teaches:
An apparatus for processing images, the apparatus comprising:
at least one memory ([0112] “Such a computer program may be stored in a non-transitory, tangible computer readable storage medium”); and
at least one processor coupled to the at least one memory ([0100] “. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user”) and configured to:
obtain scene information based on a scene ([0022] “The plurality of cameras captures a set of images of the local area. The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”; [0033] “determine depth of a scene”);
determine a depth scheme from among a plurality of depth schemes based on the scene information ([0033] The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”);
use the depth scheme to obtain depth information of the scene ([0034] The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”); and
process an image of the scene based on the depth information ([0034] “creates or updates a depth model describing the local area based on the depth information”; [0042] “The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area (e.g., the depth model), update a model of the local area, or some combination thereof”).
Regarding Claim 6, Grossinger teaches the apparatus of claim 1, wherein the scene information comprises at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
information based on an object in the scene ([0034] “depth sensing condition may be, e.g., an environmental condition of the local area (e.g., an ambient light level), a location of objects in the local area (e.g., the distance to an object from the DCA)”);
information indicative of a confidence of the depth information ([0024] “the DCA may determine a confidence map associated with the depth model and may input the depth model, the confidence map, and the set of captured images into the machine learning model. The machine learning model outputs the refined depth model”);
information indicative of motion of a device that obtains the depth information ([0095] “The position sensor 840 is an electronic device that generates data indicating a position of the headset 805. The position sensor 840 generates one or more measurement signals in response to motion of the headset 805”);
information indicative of motion within the scene;
information indicative of lighting within the scene ([0034] “the illuminator 140 illuminates a portion of the local area with light. The light may be, e.g., structured light (e.g., dot pattern, bars, etc.) in the infrared (IR), uniform light illuminating a scene, IR flash for time-of-flight, etc. In some embodiments, the one or more imaging devices 130 capture a set of images of the portion of the local area that include the light from the illuminator 140”);
information related to of the depth information ([0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”); or
tone/color/tint information related to the scene ([0082] “The example depth model 617 is depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area. The example refined depth model 642 is an example of the refined depth model 640 determined by the machine learning model 630. The example refined depth model 642 is also depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area”).
Regarding Claim 7, Grossinger teaches the apparatus of claim 6, wherein the information based on the object comprises at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
information related to a classification of the object;
information indicative of motion of the object ([0046] “The images may be analyzed together in stereo depth sensing modes to calculate distances to objects in the local area”; [0055] “The depth sensing conditions may comprise locations of objects within the local area”);
information indicative of a depth of the object in the scene ([0053] “Depth measurements may include one or more distances between a device the DCA is a component of and one or more real-world objects in the local area, one or more distances between two or more real-world objects in the local area, an orientation of the user of the device the DCA is a component of in the local area, and so on”; [0068] “calculating a depth to an object”); or
information indicative of a confidence of the information indicative of the depth of the object in the scene (Grossinger, [0060] “The depth determination module 250 determines the associated confidence map by determining a confidence value for each pixel of the corresponding depth model. For example, the depth determination module 250 compares the depth model to the set of captured images and assigns a confidence value to each pixel of the depth model based on the comparison”).
Regarding Claim 8, Grossinger teaches the apparatus of claim 1, wherein, to use a depth scheme of the plurality of depth schemes to obtain depth information, the at least one processor is configured to obtain the depth information using one or more depth modes ([0033] “The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”; [0034] “The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”), the one or more depth modes comprising at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique ([0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique ([0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity” [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 9, Grossinger teaches the apparatus of claim 1, wherein, to determine the depth scheme, the at least one processor is configured to determine to obtain the depth information based on two or more depth modes ([0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”).
Regarding Claim 12, Grossinger teaches the apparatus of claim 1, wherein the at least one processor is configured to adjust, based on the scene information, a rate at which the depth scheme determines the depth information ([0023] “The DCA may switch between depth sensing modes when preferable. The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; [0071] “The DCA may modify the depth sensing modes based on the calculated distances”).
Regarding Claim 15, Grossinger teaches:
A method for processing images, the method comprising:
obtaining scene information based on a scene ([0022] “The plurality of cameras captures a set of images of the local area. The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”; [0033] “determine depth of a scene”);
determining a depth scheme from among a plurality of depth schemes based on the scene information ([0033] The DCA controller 150 computes depth information for the portion of the local area using the set of captured images and one or more depth sensing modes”);
using the depth scheme to obtain depth information of the scene ([0034] The DCA controller 150 selects one or more depth sensing modes for the local area. The depth sensing mode may be selected based on a depth sensing condition…The DCA controller 150 obtains the depth information using the selected depth sensing modes”); and
processing an image of the scene based on the depth information ([0034] “creates or updates a depth model describing the local area based on the depth information”; [0042] “The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area (e.g., the depth model), update a model of the local area, or some combination thereof”).
Regarding Claim 20, Grossinger teaches the method of claim 15, wherein the scene information comprises at least one of (Examiners Note: since the claim recites “at least one of”, the Examiner is considering that at least one of the limitations is met by mapping the claims accordingly):
information based on an object in the scene ([0034] “depth sensing condition may be, e.g., an environmental condition of the local area (e.g., an ambient light level), a location of objects in the local area (e.g., the distance to an object from the DCA)”);
information indicative of a confidence of the depth information ([0024] “he DCA may determine a confidence map associated with the depth model and may input the depth model, the confidence map, and the set of captured images into the machine learning model. The machine learning model outputs the refined depth model”);
information indicative of motion of a device that obtains the depth information ([0095] “The position sensor 840 is an electronic device that generates data indicating a position of the headset 805. The position sensor 840 generates one or more measurement signals in response to motion of the headset 805”);
information indicative of motion within the scene;
information indicative of lighting within the scene ([0034] “the illuminator 140 illuminates a portion of the local area with light. The light may be, e.g., structured light (e.g., dot pattern, bars, etc.) in the infrared (IR), uniform light illuminating a scene, IR flash for time-of-flight, etc. In some embodiments, the one or more imaging devices 130 capture a set of images of the portion of the local area that include the light from the illuminator 140”);
information related to of the depth information ([0022] “The DCA is capable of obtaining depth information, based on the set of captured images, in multiple different modes”); or
tone/color/tint information related to the scene ([0082] “The example depth model 617 is depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area. The example refined depth model 642 is an example of the refined depth model 640 determined by the machine learning model 630. The example refined depth model 642 is also depicted with a color-scale representing different depths of objects (or of surfaces of objects) within the local area”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2 – 5, 13 – 14 and 16 - 19 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger) in view of Zobel et al. (US 20230216999 A1; hereafter referred to as Zobel).
Regarding Claim 2, Grossinger teaches the apparatus of claim 1, wherein the scene information comprises first scene information based on the scene at a first time, the depth scheme comprises a first depth scheme, the depth information comprises first depth information obtained by the first depth scheme at a second time, the image comprises a first image of the scene (Grossinger, [0055] “The depth determination module 250 may select a depth sensing mode based on one or more depth sensing conditions for regions of the local area. For example, the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes. For example, the controller 230 may obtain a first set of depth information using a structured light mode using a first camera and the illuminator”), and the at least one processor is configured to:
obtain second scene information based on the scene (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”);
determine a second depth scheme from among the plurality of depth schemes based on the second scene information, wherein the second depth scheme is different than the first depth scheme (Grossinger, [0034] “A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0067] The DCA 300 may use different depth sensing modes for different portions of the local area 310. For example, the first region 340 may be located in a shadow, and the second region 360 may be located in a well-lit area. In another example, the first region 340 may comprise a smooth wall with minimal texture, and the second region 360 may comprise multiple objects located at different depths relative to the DCA 300”);
use the second depth scheme to obtain second depth information of the scene at a fourth time (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”; Grossinger, [0055] “the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”); and
process a second image of the scene based on the second depth information (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator. The controller 230 may obtain a third set of depth information using a stereo depth sensing mode. In response to detecting a difference between expected measurements, the calibration module 260 may determine that the cameras should be calibrated, and the calibration module 260 may adjust a stereo depth matching algorithm to account for the discrepancy”).
While Grossinger teaches depth sensing conditions based on time of the day measurements which is broadly considered as different times, it fails to explicitly teach:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time,
obtain second scene information based on the scene at a third time;
use the second depth scheme to obtain second depth information of the scene at a fourth time;
In the same field of endeavor, Zobel teaches:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time (Zobel, [0232] “the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time….the first image data includes at least one image depicting the environment at least at one of the first time”),
obtain second scene information based on the scene at a third time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
use the second depth scheme to obtain second depth information of the scene at a fourth time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
Grossinger and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Zobel to make the invention that obtains first depth information at a second time; obtains second scene information based on the scene at a third time and obtains second depth information of the scene at a fourth time; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 3, Grossinger in view of Zobel teaches the apparatus of claim 2, wherein:
to use the first depth scheme of the plurality of depth schemes to obtain the depth information, the at least one processor is configured to obtain the depth information using a first number of depth modes of a plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”); and
to use the second depth scheme of the plurality of depth schemes to obtain depth information, the at least one processor is configured to obtain the depth information using a second number of depth modes of the plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”).
Regarding Claim 4, Grossinger in view of Zobel teaches the apparatus of claim 3, wherein the plurality of depth modes comprises at least two of (Examiners Note: since the claim recites “at least two of”, the Examiner is considering that at least two of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique (Grossinger, [0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique (Grossinger, [0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity” [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 5, Grossinger in view of Zobel teaches the apparatus of claim 3, wherein the first number of depth modes is different from the second number of depth modes (Grossinger, [0023] “The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes).
Regarding Claim 13, Grossinger in view of Zobel teaches the apparatus of claim 12, but fails to explicitly teach:
wherein the at least one processor is configured to interpolate between instances of depth information to generate interpolated depth information.
In the same field of endeavor, Zobel teaches:
wherein the at least one processor is configured to interpolate between instances of depth information to generate interpolated depth information (Zobel, [0079] “ The change in perspective can be used for frame interpolation to increase effective frame rate of a video by generating an intermediate frame in between two existing frames”; Zobel, [0134] “the smaller motion vector maps generated by the time warp 705 can be used to interpolate additional frames in between existing frames of a video, for instance to increase the frame rate of the video from a first frame rate to a second frame rate that is higher than the first frame rate”).
Grossinger and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Zobel to make the invention that interpolates between instances of depth information to generate interpolated depth information; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 14, Grossinger in view of Zobel teaches the apparatus of claim 12, wherein the rate at which the depth scheme determines the depth information (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes) is adjusted separately from an image-capture rate (Zobel, [0079] “The change in perspective can be used for frame interpolation to increase effective frame rate of a video by generating an intermediate frame in between two existing frames”; Zobel, [0134] “the smaller motion vector maps generated by the time warp 705 can be used to interpolate additional frames in between existing frames of a video, for instance to increase the frame rate of the video from a first frame rate to a second frame rate that is higher than the first frame rate”).
Grossinger and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Zobel to make the invention that interpolates between instances of depth information to generate interpolated depth information; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 16, Grossinger teaches the method of claim 15, wherein the scene information comprises first scene information based on the scene at a first time, the depth scheme comprises a first depth scheme, the depth information comprises first depth information obtained by the first depth scheme at a second time, the image comprises a first image of the scene (Grossinger, [0055] “The depth determination module 250 may select a depth sensing mode based on one or more depth sensing conditions for regions of the local area. For example, the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes. For example, the controller 230 may obtain a first set of depth information using a structured light mode using a first camera and the illuminator”), and the at least one processor is configured to:
obtaining second scene information based on the scene (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”); (Zobel, [0232] “the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time….the first image data includes at least one image depicting the environment at least at one of the first time”)
determining a second depth scheme from among the plurality of depth schemes based on the second scene information, wherein the second depth scheme is different than the first depth scheme (Grossinger, [0034] “A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0067] The DCA 300 may use different depth sensing modes for different portions of the local area 310. For example, the first region 340 may be located in a shadow, and the second region 360 may be located in a well-lit area. In another example, the first region 340 may comprise a smooth wall with minimal texture, and the second region 360 may comprise multiple objects located at different depths relative to the DCA 300”);
using the second depth scheme to obtain second depth information of the scene at a fourth time (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator”; Grossinger, [0055] “the depth sensing conditions may include an expected ambient light level. The expected ambient light level may be based on previously observed ambient light levels for the local area, geographic information about the local area, a time of day of the depth measurement”); and
processing a second image of the scene based on the second depth information (Grossinger, [0064] “The controller 230 may obtain a second set of depth information using a structured light mode using a second camera and the illuminator. The controller 230 may obtain a third set of depth information using a stereo depth sensing mode. In response to detecting a difference between expected measurements, the calibration module 260 may determine that the cameras should be calibrated, and the calibration module 260 may adjust a stereo depth matching algorithm to account for the discrepancy”).
While Grossinger teaches depth sensing conditions based on time of the day measurements which is broadly considered as different times, it fails to explicitly teach:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time,
obtaining second scene information based on the scene at a third time;
using the second depth scheme to obtain second depth information of the scene at a fourth time;
In the same field of endeavor, Zobel teaches:
the scene information comprises first scene information based on the scene at a first time, the depth information comprises first depth information obtained by the first depth scheme at a second time (Zobel, [0232] “the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time….the first image data includes at least one image depicting the environment at least at one of the first time”),
obtaining second scene information based on the scene at a third time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
using the second depth scheme to obtain second depth information of the scene at a fourth time (Zobel, [0292] “wherein the second image data includes an interpolated image configured to depict the environment at a second time”);
Grossinger and Zobel are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Zobel to make the invention that obtains first depth information at a second time; obtains second scene information based on the scene at a third time and obtains second depth information of the scene at a fourth time; doing so can provide depth information of the scene in different perspectives at different times to depict the environment (Zobel, [0005]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 17, Grossinger in view of Zobel teaches the method of claim 16, wherein:
using the first depth scheme of the plurality of depth schemes to obtain the depth information, the at least one processor is configured to obtain the depth information using a first number of depth modes of a plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”); and
using the second depth scheme of the plurality of depth schemes to obtain depth information, the at least one processor is configured to obtain the depth information using a second number of depth modes of the plurality of depth modes (Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes. The DCA controller 150 creates or updates a depth model describing the local area based on the depth information. A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”).
Regarding Claim 18, Grossinger in view of Zobel teaches the method of claim 17 wherein the plurality of depth modes comprises at least two of (Examiners Note: since the claim recites “at least two of”, the Examiner is considering that at least two of the limitations is met by mapping the claims accordingly):
a phase-detection depth-determination technique;
a monocular depth-determination technique;
a machine-learning-model-based depth-determination technique (Grossinger, [0024] “the DCA may utilize a machine learning model to update the depth model by generating a refined depth model”);
a depth-from-stereo depth-determination technique; or an active-illumination depth-determination technique (Grossinger, [0023] “The DCA dynamically determines depth sensing modes (e.g., passive stereo, active stereo, structured stereo) based in part on the surrounding environment and/or user activity” [0033] “The depth sensing mode may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator 140), some other mode to determine depth of a scene, or some combination thereof”).
Regarding Claim 19, Grossinger in view of Zobel teaches the method of claim 17, wherein the first number of depth modes is different from the second number of depth modes (Grossinger, [0023] “The DCA may use different depth sensing modes for different portions of the local area. The DCA uses the depth information to update a depth model describing the local area”; Grossinger, [0034] “The DCA controller 150 obtains the depth information using the selected depth sensing modes...A portion of the depth model may be obtained using a first depth sensing mode, and different portion of the depth model may be obtained using a second depth sensing mode”; Grossinger, [0064] “The controller 230 may obtain depth information for a region using multiple depth sensing modes).
Claims 10 – 11 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US 20210366142 A1; hereafter referred to as Grossinger) in view of Wantland et al. (US 20210042950 A1; hereafter referred to as Wantland).
Regarding Claim 10, Grossinger teaches the apparatus of claim 1, but fails to explicitly teach:
wherein the at least one processor is configured to modify the image of the scene based on the depth information.
In the same field of endeavor, Wantland teaches:
wherein the at least one processor is configured to modify the image of the scene based on the depth information (Wantland, [0079] “By utilizing a segmentation mask or masks in combination with depth information for the same image or images, an example graphic-object addition process may allow for augmented-reality style photo editing, where virtual objects are generated and/or modified so as to more realistically interact with the real-world objects in the image or images”).
Grossinger and Wantland are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Grossinger with the invention of Wantland to make the invention that modifies the image of the scene based on the depth information; doing so can provide depth information of edited version of the image wherein corrections are made to remove artifacts (Wantland, [0002]- [0003]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 11, Grossinger in view of Wantland teaches the apparatus of claim 10, wherein the at least one processor is configured to:
identify foreground pixels of the image based on the depth information, wherein the foreground pixels represent a foreground of the scene (Wantland, [0019] “an example mask may involve setting all pixels that correspond to an object in the foreground of an image to white and all pixels that correspond to background features or objects to black”; Wantland, [0025] “the depth map can include a depth value for each pixel in an image”); and
identify background pixels of the image based on the depth information, wherein the background pixels represent a background of the scene (Wantland, [0034] To capture dual pixels, the camera can use a sensor that captures two slightly different views of a scene. In comparing these two views, a foreground object can appear to be stationary while background objects move vertically in an effect referred to as parallax. For example, a “selfie” or image of a person taken by that person typically has the face of that person as a foreground object and may have other objects in the background”; Wantland, [0068] “A depth map may also be utilized to determine first depth information for the at least one subject, and second depth information for the at least one background area. The perspective adjustment process may then compare the first and second depth information to determine an amount of movement for a background area in the image frame, per unit of movement of at least one subject in the image frame”);
wherein the image is modified based on the foreground pixels and the background pixels (Wantland, [0096] “segmentation masks may be used to separate the background of an image from objects in the foreground of the image. Depth information for the background may then be utilized to determine an amount of blurring to be applied to the background; Wantland, [0098] “segmentation data for image 801 provides segmentation masks for at least foreground objects 806 (e.g., the table, lamp, picture, and plant), such that the background 804 can be separated from the foreground objects 806. Further, once the background 804 is separated from the foreground objects 806, a depth map for image 801 can be used to determine depth information specifically for the background 804”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220156426 A1 SCENE LAYOUT ESTIMATION
US 20210241495 A1 METHOD AND SYSTEM FOR RECONSTRUCTING COLOUR AND DEPTH INFORMATION OF A SCENE
US 9282313 B2 Methods And Systems For Converting 2D Motion Pictures For Stereoscopic 3D Exhibition
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664