DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Simonetto et al. (Lightweight Deep Learning Architecture for MPI Correction and Transient Reconstruction, 29 November 2021, arXiv, Pages 1-11), hereinafter “Simonetto”, as cited in the IDS filed 11 February 2025.
Simonetto anticipates each of claims 1-18. This reference is prior art under 35 U.S.C. 102(a)(1) since it falls within the one year grace period but is not to only one or more joint inventors, due to the addition of “Zanuttigh” and “Schäfer”. However, a declaration under Rule 130(a) may be filed to disqualify the reference under 102(b)(1)(A).
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-6, 8, 11, 12, 17, and 18 is/are also rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Gil-Cacho (U.S. Pub. No. 2021/0231812), as cited in the IDS filed 11 February 2025.
Regarding claim 1, Gil-Cacho teaches:
A method comprising applying a machine learning model-based regression to a phasor image captured by an iToF sensor or phasor data obtained from the phasor image (See [0112]: “The deep neural network may use a convolutional architecture and may be trained for regression, i.e., to perform jointly direct component and global component separation and an inpainting by using models and priors learned from a dataset.”) with spot illumination (See [0055]: “The high intensity light areas 5 are in a form of multiple dots patterns that impose a sparse sampling grid on the scene 3, for example, dots are centered at the coordinates lit up by each spot.”) to obtain an estimate of the direct light component of the phasor image and/or an estimate of the global light component of the phasor image (See [0113]: “Moreover, the deep neural network may output a high-resolution direct component image data.”).
Regarding claim 2, Gil-Cacho teaches:
The method of claim 1, wherein the phasor data is single frequency spot-iToF data (See In some embodiments, the I and Q values (i.e. of the corresponding (first/second/direct component) image data) may be used and a phase value may be estimated for the corresponding (first/second/direct component) image data), for example, based on the following equation:”.).
Regarding claim 3, Gil-Cacho teaches:
The method of claim 1, wherein the machine learning based model regression is applied to the phasor image to obtain an estimate of the global component of the phasor image (See [0112]: “The deep neural network may use a convolutional architecture and may be trained for regression, i.e., to perform jointly direct component and global component separation and an inpainting by using models and priors learned from a dataset.”), and wherein the method further comprises determining an estimate of the direct component based on the estimate of the global component and based on a phasor image (See [0090]: “In some embodiments, the direct component image data may be representative of a signal when the direct component of the light is separated from the global component of the light, without limiting the present disclosure to any specific embodiment.”).
Regarding claim 4, Gil-Cacho teaches:
The method of claim 1, wherein the machine learning based model regression in addition to the phasor image captured by an iToF sensor or in addition to the phasor data obtained from the phasor image takes auxiliary data as further input (See [0112]: “For example, the first image data (i.e. dots areas), the second image data (i.e. valleys areas), the intensity map, the first mask and the second mask may be fed into the deep neural network.”).
Regarding claim 5, Gil-Cacho teaches:
The method of claim 4, wherein the auxiliary data are data from other modes and frequencies, a full-frame infrared or grayscale image sampled by the same sensor (See the intensity map in [0112].) or phasor images at higher or lower frequencies than the reference one.
Regarding claim 6, Gil-Cacho teaches:
The method of claim 4, wherein the auxiliary data is a multi-channel image that stacks data from different channels (See the first mask and second mask in [0112].).
Regarding claim 8, Gil-Cacho teaches:
The method of claim 1, wherein the estimate of the direct component and the estimate of the global component are sparse phasor images describing the direct and global components at the centers of the sparse spot illumination (See [0055]: “The high intensity light areas 5 are in a form of multiple dots patterns that impose a sparse sampling grid on the scene 3, for example, dots are centered at the coordinates lit up by each spot.” The examiner asserts that the direct and global estimates from this sparse sampling grid are also sparse.).
Regarding claim 11, Gil-Cacho teaches:
A method for training a machine learning model for direct and global light component regression, the method comprising generating training data comprising a direct ground truth phasor and/or a global ground truth phasor based on a 3D model/scene (See [0112]: “The deep neural network may use a convolutional architecture and may be trained for regression, i.e., to perform jointly direct component and global component separation and an inpainting by using models and priors learned from a dataset.”).
Regarding claim 12, Gil-Cacho teaches:
The method of claim 11, wherein the method for training a machine learning-based regression model further comprises training the machine learning-based regression model based on the direct ground truth phasor and/or the global ground truth phasor (See the deep neural network that is trained for regression in [0112].).
Gil-Cacho teaches claim 17 for the reasons given in the treatment of claim 1.
Gil-Cacho teaches claim 18 for the reasons given in the treatment of claim 11.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 7, 9, and 13-16 is/are also rejected under 35 U.S.C. 103 as being unpatentable over Gil-Cacho (U.S. Pub. No. 2021/0231812) in view of Buratto et al. (Deep Learning for Transient Image Reconstruction from ToF Data, 11 March 2021, Sensors, Vol. 21, No. 1962, Pages 1-20), hereinafter “Buratto”, as cited in the IDS filed 11 February 2025.
Claim 7 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 1, wherein
Gil-Cacho does not explicitly disclose the following; however, Buratto discloses:
the machine learning-based model regression is pretrained based on one or more ground truth images obtained based on direct/global separation of transient image of a model scene (See page 9: “In order to get a meaningful result we therefore make use of a reconstruction loss, which simply ensures that our prediction ˆx matches with the ground truth x.” Then see page 10: “For the supervised optimization of the proposed approach we need a training set containing raw ToF data together with the corresponding ground truth transient data.”).
Gil-Cacho and Buratto together disclose the limitations of claim 7. Buratto is directed to a similar field of art (estimation of direct and global components of incoming ToF data). Therefore, Gil-Cacho and Buratto are combinable. Gil-Cacho does not explicitly mention pretraining (1) using ground truth images and (2) of which are based on separation of a transient image. Modifying the system and method of Gil-Cacho by adding the capability of “[pretraining] based on one or more ground truth images obtained based on direct/global separation of transient image of a model scene”, as taught by Buratto, would yield the expected and predictable result of improved predictions by the model. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine Gil-Cacho and Buratto in this way.
Claim 9 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 8, wherein the method further comprises
Gil-Cacho does not explicitly disclose the following; however, Buratto discloses:
performing a concatenation on one or more neighborhoods of the phasor image to obtain the phasor data (See page 7: “where v ∈ CM×1 is the stack of the raw camera measurements in the complex domain at different modulation frequencies”. The examiner asserts that stacking the raw camera measurements into a vector is a form of concatenation.).
See the motivation to combine in the treatment of claim 7.
Claim 10 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 1, wherein
Gil-Cacho does not disclose the following; however, Buratto discloses:
the estimate of the direct component and the estimate of the global component are dense phasor images describing the direct and global components at the full resolution of the iToF sensor (See page 3: “Transient cameras are relatively new devices that do exactly this. In practice, transient sensors are able to capture the incoming intensity of light at extremely high temporal resolutions. As we are working with the speed of light, current sensors need a temporal resolution in the order of the tens of picoseconds [25] for millimeter distance resolution”).
See the motivation to combine in the treatment of claim 7.
Claim 13 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 11, wherein the method for training a machine learning-based regression model further comprises
Gil-Cacho does not disclose the following; however, Buratto discloses:
determining a transient image from the 3D model/scene (See page 6: “The main novelty we propose is the introduction of the transient information inside our training pipeline.”).
See the motivation to combine in the treatment of claim 7.
Claim 14 is met by the combination of Gil-Cacho and Buratto, wherein
The combination of Gil-Cacho and Buratto teaches:
The method of claim 13, wherein the method for training a machine learning-based regression model further comprises
And Buratto further discloses:
applying an iToF sensor model and optics on the transient image (See page 6: “Starting from Equation (8), and following the study done in [36], we will express the relation between raw iToF measurements and the corresponding backscattering into a simple matrix multiplication. Following that, we will introduce our model which takes in input raw iToF measurements in order to predict transient information.”).
See the motivation to combine in the treatment of claim 7.
Claim 15 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 11, wherein the method for training a machine learning-based regression model further comprises
Gil-Cacho does not disclose the following; however, Buratto discloses:
applying a direct/global separation to a transient image to obtain the direct ground truth phasor and/or the global ground truth phasor (See page 4: “The deep learning pipeline we propose is split into two main blocks: a predictive model, which learns the relation between the noisy iToF measurements and the encoded version of our transient data and a fixed model which translates the encoded information into the corresponding transient vector. While the network was developed under the strongly simplifying assumption that MPI is related to a backscattering vector composed by two peaks, one for the direct light reflection and a second peak summarizing the global light”. Then see page 7: “In practice however, since our task is MPI denoising, we really just need an accurate localization of the direct component (the first peak), while for the second global component a more concise encoding can suffice.”).
See the motivation to combine in the treatment of claim 7.
Claim 16 is met by the combination of Gil-Cacho and Buratto, wherein
Gil-Cacho teaches:
The method of claim 11, wherein the method for training a machine learning-based regression model further comprises
Gil-Cacho does not disclose the following; however, Buratto discloses:
illuminating the 3D model/scene by an illumination profile and rendering the 3D model/scene by a transient renderer to obtain a transient image (See page 18: “Our work leaves open several future research directions, first of all an extension of the backscattering model which as of now is quite simple and an extension of the approach employing a similar model for transient data reconstruction.”).
See the motivation to combine in the treatment of claim 7.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN S LEE whose telephone number is (571)272-1981. The examiner can normally be reached 11:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jonathan S Lee/Primary Examiner, Art Unit 2677
Prosecution Insights