DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claim 2 recites a limitation of the form “at least one of A and B”. In accordance with the U.S. Court of Appeals for the Federal Circuit in SuperGuide Corp v. DirecTV Enterprises, Inc., these limitations are conjunctive in nature and to be construed as “at least one of A and at least one of B”. Therefore, these claims are addressed herein as requiring each of these steps rather than the alternative of A or B. Similar reasoning applies to claim 11.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7, 10-12, 14, 16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Sunkavalli et al. (US 2019/0164312) and Huang et al. (“Normalization Techniques in Training DNNs: Methodology, Analysis and Application”).
Regarding claim 1, Sunkavalli et al. discloses a computer-implemented method for training a machine-learned model, the method comprising:
obtaining, by a processor, a plurality of images, wherein a plurality of sets of parameter values are respectively associated with the plurality of images, each set of parameter values comprising values for a plurality of camera parameters (“Referring now to FIG. 2, a block diagram is provided illustrating an exemplary system 200 for training a convolutional neural network that can determine camera calibration parameters corresponding to a camera that captured a digital image, and for implementing or configuring the convolutional neural network to determine the camera calibration parameters of the digital image. Although the described embodiments are included with a focus on extrinsic camera parameters, it is understood that some intrinsic parameters (e.g., focal length) can also be determined in accordance with some embodiments described herein. It is noted that a relationship between intrinsic and extrinsic parameters of a digital image, described in detail herein, is employable to facilitate various embodiments of the present disclosure and, to this end, the system 200 can also be described as one that determines at least some of the intrinsic camera parameters for a camera that captured a digital image” at paragraph 0027).
Sunkavalli et al. does not explicitly disclose determining, by the processor, a covariance matrix for the plurality of camera parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values, performing, by the processor, a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix and performing, by the processor, an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass and applying the preconditioning matrix in a backward gradient pass.
Huang et al. teaches a method for training a machine-learned model, the method comprising:
determining, by the processor, a covariance matrix for the plurality of parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values (“Decorrelating formulates the transformation as: ˆx = ΦD(x) = Dx, (11) where D = [d1, . . . , dd] are the eigenvectors of Σ and Σ = ED(xxT ) is the covariance matrix” at page 10175, right column, line 5);
performing, by the processor, a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix (“Whitening formulates the transformation as1: equation (13) where ˜Λ = diag(˜σ1, . . . , ˜σd) and D = [d1, . . . , dd] are the eigenvalues and associated eigenvectors of covariance matrix Σ. Whitening ensures that the normalized output ˆx has a spherical Gaussian distribution, which can be represented as: ED(ˆxˆxT) = I. The whitening transformation, defined in (13), is called principal components analysis (PCA) whitening, where the whitening matrix Σ −1 2 PCA = ˜Λ −1 2 d D” at page 10175, right column, line 17; “whitening can ensure the covariance matrix to be an identity matrix” at section IXB, paragraph 2, line 3); and
performing, by the processor, an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises:
applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass (“Zero-phase component analysis (ZCA) whitening, using Σ −1 2 ZCA = D˜Λ−1 2DT , is advocated for in [36], where the PCAwhitened input is rotated back by the corresponding rotation matrix D” at section V-A, 2) Normalization Operation, paragraph 3, line 9); and
applying the preconditioning matrix in a backward gradient pass (“backpropagate through the inverse square root of a matrix (i.e. ∂Σ−1 2 /∂Σ)” at section V-A, 2) Normalization Operation, paragraph 3, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the particular training as taught by Huang et al. for the machine learning of Sunkavalli et al. to “improve the trade-off between efficiency and performance” (Huang et al. at section X, line 7).
Regarding claim 10, Sunkavalli et al. discloses a computing system, comprising:
a processor (“one or more processors 814” at paragraph 0067, line 3); and
a non-transitory, computer-readable medium that, when executed by the processor, causes the processor to perform operations (“Computing device 800 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 800 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.” at paragraph 0068, line 1), the operations comprising:
obtaining a plurality of images, wherein a plurality of sets of parameter values are respectively associated with the plurality of images, each set of parameter values comprising values for a plurality of camera parameters (“Referring now to FIG. 2, a block diagram is provided illustrating an exemplary system 200 for training a convolutional neural network that can determine camera calibration parameters corresponding to a camera that captured a digital image, and for implementing or configuring the convolutional neural network to determine the camera calibration parameters of the digital image. Although the described embodiments are included with a focus on extrinsic camera parameters, it is understood that some intrinsic parameters (e.g., focal length) can also be determined in accordance with some embodiments described herein. It is noted that a relationship between intrinsic and extrinsic parameters of a digital image, described in detail herein, is employable to facilitate various embodiments of the present disclosure and, to this end, the system 200 can also be described as one that determines at least some of the intrinsic camera parameters for a camera that captured a digital image” at paragraph 0027).
Sunkavalli et al. does not explicitly disclose determining a covariance matrix for the plurality of camera parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values, performing a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix and performing an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass and applying the preconditioning matrix in a backward gradient pass.
Huang et al. teaches a computing system, comprising:
a processor (processor of implied computer); and
a non-transitory, computer-readable medium that, when executed by the processor, causes the processor to perform operations (implied that the computer is programmed accordingly), the operations comprising:
determining a covariance matrix for the plurality of parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values (“Decorrelating formulates the transformation as: ˆx = ΦD(x) = Dx, (11) where D = [d1, . . . , dd] are the eigenvectors of Σ and Σ = ED(xxT ) is the covariance matrix” at page 10175, right column, line 5);
performing a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix (“Whitening formulates the transformation as1: equation (13) where ˜Λ = diag(˜σ1, . . . , ˜σd) and D = [d1, . . . , dd] are the eigenvalues and associated eigenvectors of covariance matrix Σ. Whitening ensures that the normalized output ˆx has a spherical Gaussian distribution, which can be represented as: ED(ˆxˆxT) = I. The whitening transformation, defined in (13), is called principal components analysis (PCA) whitening, where the whitening matrix Σ −1 2 PCA = ˜Λ −1 2 d D” at page 10175, right column, line 17; “whitening can ensure the covariance matrix to be an identity matrix” at section IXB, paragraph 2, line 3); and
performing an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises:
applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass (“Zero-phase component analysis (ZCA) whitening, using Σ −1 2 ZCA = D˜Λ−1 2DT , is advocated for in [36], where the PCAwhitened input is rotated back by the corresponding rotation matrix D” at section V-A, 2) Normalization Operation, paragraph 3, line 9); and
applying the preconditioning matrix in a backward gradient pass (“backpropagate through the inverse square root of a matrix (i.e. ∂Σ−1 2 /∂Σ)” at section V-A, 2) Normalization Operation, paragraph 3, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the particular training as taught by Huang et al. for the machine learning of Sunkavalli et al. to “improve the trade-off between efficiency and performance” (Huang et al. at section X, line 7).
Regarding claim 19, Sunkavalli et al. discloses a non-transitory, computer-readable medium that, when executed by a processor (“one or more processors 814” at paragraph 0067, line 3), causes the processor to perform operations (“Computing device 800 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 800 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.” at paragraph 0068, line 1), the operations comprising:
obtaining a plurality of images, wherein a plurality of sets of parameter values are respectively associated with the plurality of images, each set of parameter values comprising values for a plurality of camera parameters (“Referring now to FIG. 2, a block diagram is provided illustrating an exemplary system 200 for training a convolutional neural network that can determine camera calibration parameters corresponding to a camera that captured a digital image, and for implementing or configuring the convolutional neural network to determine the camera calibration parameters of the digital image. Although the described embodiments are included with a focus on extrinsic camera parameters, it is understood that some intrinsic parameters (e.g., focal length) can also be determined in accordance with some embodiments described herein. It is noted that a relationship between intrinsic and extrinsic parameters of a digital image, described in detail herein, is employable to facilitate various embodiments of the present disclosure and, to this end, the system 200 can also be described as one that determines at least some of the intrinsic camera parameters for a camera that captured a digital image” at paragraph 0027).
Sunkavalli et al. does not explicitly disclose determining a covariance matrix for the plurality of camera parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values, performing a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix and performing an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass and applying the preconditioning matrix in a backward gradient pass.
Huang et al. teaches a non-transitory, computer-readable medium that, when executed by a processor (processor of implied computer), causes the processor to perform operations implied that the computer is programmed accordingly), the operations comprising:
determining a covariance matrix for the plurality of parameters with respect to a plurality of projected points generated via evaluation of a projection function at the plurality of sets of parameter values (“Decorrelating formulates the transformation as: ˆx = ΦD(x) = Dx, (11) where D = [d1, . . . , dd] are the eigenvectors of Σ and Σ = ED(xxT ) is the covariance matrix” at page 10175, right column, line 5);
performing a whitening algorithm to identify a preconditioning matrix that, when applied to the plurality of sets of parameter values, results in the covariance matrix being approximately equal to an identity matrix (“Whitening formulates the transformation as1: equation (13) where ˜Λ = diag(˜σ1, . . . , ˜σd) and D = [d1, . . . , dd] are the eigenvalues and associated eigenvectors of covariance matrix Σ. Whitening ensures that the normalized output ˆx has a spherical Gaussian distribution, which can be represented as: ED(ˆxˆxT) = I. The whitening transformation, defined in (13), is called principal components analysis (PCA) whitening, where the whitening matrix Σ −1 2 PCA = ˜Λ −1 2 d D” at page 10175, right column, line 17; “whitening can ensure the covariance matrix to be an identity matrix” at section IXB, paragraph 2, line 3); and
performing an optimization algorithm on the plurality of sets of parameter values, wherein performing the optimization algorithm comprises:
applying an inverse of the preconditioning matrix to the plurality of sets of parameters in a forward prediction pass (“Zero-phase component analysis (ZCA) whitening, using Σ −1 2 ZCA = D˜Λ−1 2DT , is advocated for in [36], where the PCAwhitened input is rotated back by the corresponding rotation matrix D” at section V-A, 2) Normalization Operation, paragraph 3, line 9); and
applying the preconditioning matrix in a backward gradient pass (“backpropagate through the inverse square root of a matrix (i.e. ∂Σ−1 2 /∂Σ)” at section V-A, 2) Normalization Operation, paragraph 3, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the particular training as taught by Huang et al. for the machine learning of Sunkavalli et al. to “improve the trade-off between efficiency and performance” (Huang et al. at section X, line 7).
Regarding claims 2 and 11, Sunkavalli et al. discloses a method and system wherein the plurality of camera parameters includes at least one of an intrinsic parameter and an extrinsic parameter (“Referring now to FIG. 2, a block diagram is provided illustrating an exemplary system 200 for training a convolutional neural network that can determine camera calibration parameters corresponding to a camera that captured a digital image, and for implementing or configuring the convolutional neural network to determine the camera calibration parameters of the digital image. Although the described embodiments are included with a focus on extrinsic camera parameters, it is understood that some intrinsic parameters (e.g., focal length) can also be determined in accordance with some embodiments described herein. It is noted that a relationship between intrinsic and extrinsic parameters of a digital image, described in detail herein, is employable to facilitate various embodiments of the present disclosure and, to this end, the system 200 can also be described as one that determines at least some of the intrinsic camera parameters for a camera that captured a digital image” at paragraph 0027).
Regarding claims 3 and 12, the Sunkavalli et al. and Huang et al. combination discloses the elements of claims 2 and 11 above.
The Sunkavalli et al. and Huang et al. combination does not explicitly disclose that two or more sets of parameter values share at least one intrinsic parameter value of the plurality of camera parameters.
However, given that there are a multitude of training sets, it is feasible that at least two sets have at least one intrinsic parameter, such as focal length, in common. As such, this is an obvious element in view of the disclosure of the Sunkavalli et al. reference.
Regarding claims 5 and 14, Huang et al. discloses a method and system wherein the whitening algorithm is a whitening algorithm selected from a group of whitening algorithms consisting of principal component analysis, zero component analysis (“Zero-phase component analysis (ZCA) whitening, using Σ −1 2 ZCA = D˜Λ−1 2DT , is advocated for in [36], where the PCAwhitened input is rotated back by the corresponding rotation matrix D” at section V-A, 2) Normalization Operation, paragraph 3, line 9), and canonical correlation analysis.
Regarding claims 7 and 16, the Sunkavalli et al. and Huang et al. combination discloses a method and system wherein at least one value not on a diagonal in the covariance matrix represents a correlation of motion between a first camera parameter and a second camera parameter (“Decorrelating formulates the transformation as: ˆx = ΦD(x) = Dx, (11) where D = [d1, . . . , dd] are the eigenvectors of Σ and Σ = ED(xxT ) is the covariance matrix” Huang et al. at page 10175, right column, line 5).
Allowable Subject Matter
Claims 4, 6, 8, 9, 13, 15, 17, 18 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not teach or disclose that the at least one intrinsic parameter value shared by the two or more sets of parameter values is represented by an additional loss term in the optimization algorithm as required by claims 4 and 13; that at least one diagonal of the covariance matrix represents an average motion magnitude induced by varying a first parameter of the plurality of camera parameters as required by claims 6 and 15; that the covariance matrix includes a dampening parameter as required by claims 8, 17 and 20.
Prior Art Notes
Park et al. (“CamP: Camera Preconditioning for Neural Radiance Fields”) is cited as a relevant reference, but is disqualified under 35 USC 102(b)(1)(A) as it shares the same inventive entity as the instant application and is filed within the grace period.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672