Response to Arguments
Applicant’s arguments submitted on 6/17/2026 have been fully considered, but are not persuasive. Applicant argues that the prior art does not disclose adjusting the second image nor applying a registration parameter estimator. Examiner respectfully disagrees.
Javan Roshtkhari discloses adjusting the second image (see Javan Roshtkhari Figs. 3 and 5, and paras. 0027-0029, where “[t]he reference object 14 (i.e. image, or template) of the sport field model is warped according to this initial estimate”); and applying a registration parameter estimator to the adjusted second image, or both the first image and adjusted second image to: map pixel data of the first image or adjusted second image to the first image to generate an estimation of parametric registration transformation, wherein the registration parameter estimator is a trained machine learning or artificial intelligence model that has been trained using images to estimate one or more parameters of the parametric registration transformation (see Javan Roshtkhari Figs. 3 and 5, and paras. 0007, 0027-0029, and 0042-0048, where “[g]iven an input image 12, the initial camera pose, or camera parameters, represented here by a homography transformation, h, 22, are obtained using a function approximation technique shown as a DNN 16 that regresses the images directly to the homography parameterization” then “[t]he error estimation process 18 takes the warped template of the input image and estimates the misalignment error between the two” then “[t]he estimated error is then used as an objective function for iteratively optimizing the transformation parameters 22 and to update the parameters to maximize the alignment between the image 12 and the reference object 14 (template)” and “. . . i.e. the error measurement 18, which can be chosen to be a NN” (with Fig. 3 illustrating 18 as a “DNN”) and the training of the neural networks can be done “jointly” and use images).
Specifically, Fig. 3 of Javan Roshtkhari shows an image (14) is adjusted by initial warp parameters h (22) followed by a DNN registration parameter estimator (18) iteratively updating those warp parameters h (22) to improve the registration between images (12) and (14).
PNG
media_image1.png
508
875
media_image1.png
Greyscale
Claim Rejections - 35 USC § 112
The rejection under this statute is hereby withdrawn in response to Applicant’s amendments.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 11-22, 24, and 25 is/are rejected under 35 U.S.C. 102(a)(1) and 35 U.S.C. 102(a)(2) as being anticipated by Javan Roshtkhari et al., US 2020/0372679 A1 (hereinafter referred to as “Javan Roshtkhari”).
Regarding claim 1, Javan Roshtkhari discloses a method (see Javan Roshtkhari paras. 0002 and 0009-0011, where the input images are received from a camera, and the algorithm is implemented as a method, computer readable medium, and/or electronic device comprising a processor and memory) for registering images to each other or registering images to templates to generate geometric or nonlinear registration transformation mappings (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0021, where images/templates are registered by “learning a mapping function” that “can be either a geometric transformation or a non-linear transformation”), the method comprising: obtaining a first image from an imaging device, wherein the first image is in a first modality (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0052, where the first image is “sports images”); providing a second image in a second modality (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0052, where the mapping is to “templates”); adjusting the second image (see Javan Roshtkhari Figs. 3 and 5, and paras. 0027-0029, where “[t]he reference object 14 (i.e. image, or template) of the sport field model is warped according to this initial estimate”); applying a registration parameter estimator to the adjusted second image, or both the first image and adjusted second image to: map pixel data of the first image or adjusted second image to the first image to generate an estimation of parametric registration transformation, wherein the registration parameter estimator is a trained machine learning or artificial intelligence model that has been trained using images to estimate one or more parameters of the parametric registration transformation (see Javan Roshtkhari Figs. 3 and 5, and paras. 0007, 0027-0029, and 0042-0048, where “[g]iven an input image 12, the initial camera pose, or camera parameters, represented here by a homography transformation, h, 22, are obtained using a function approximation technique shown as a DNN 16 that regresses the images directly to the homography parameterization” then “[t]he error estimation process 18 takes the warped template of the input image and estimates the misalignment error between the two” then “[t]he estimated error is then used as an objective function for iteratively optimizing the transformation parameters 22 and to update the parameters to maximize the alignment between the image 12 and the reference object 14 (template)” and “. . . i.e. the error measurement 18, which can be chosen to be a NN” (with Fig. 3 illustrating 18 as a “DNN”) and the training of the neural networks can be done “jointly” and use images); and providing output data comprising one or more parameters of the parametric registration transformation (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0025, where “[t]he module 10 generates a set of one or more output parameters 22, which can include camera parameters, or parameters of the geometric and non-linear transformation, which are numerical values of the intrinsic and extrinsic camera parameters or a subset of them”).
Claims 21 and 22 are rejected under the same analysis as claim 1 above.
Regarding claim 2, Javan Roshtkhari discloses wherein the second image is either: a reference image or a template, where the reference image or the template comprises a partial or full representation of contents of the first image; or generated from the first image (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0052, where the mapping is to “templates”).
Regarding claim 3, Javan Roshtkhari discloses wherein the first and second modalities are different modalities (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009, 0022, and 0052, where images and templates being registered and/or “registering images of different image modalities” is disclosed).
Regarding claim 11, Javan Roshtkhari discloses further comprising generating a quantitative value measuring a quality of the estimation of the registration transformation (see Javan Roshtkhari para. 0020, where “an error function that estimates the registration error between the received image and the template”).
Regarding claim 12, Javan Roshtkhari discloses initializing another image registration technique for further improving a quality of the estimation of the registration transformation with parameters of the registration transformation (see Javan Roshtkhari para. 0020, where “an optimization process to iteratively update the homography transformation parameters to minimize the estimated error”).
Regarding claim 13, Javan Roshtkhari discloses wherein the adjusted second image results in an optimal or suboptimal registration (see Javan Roshtkhari Figs. 3 and 5, and paras. 0027-0029, where “[t]he reference object 14 (i.e. image, or template) of the sport field model is warped according to this initial estimate”).
Regarding claim 14, Javan Roshtkhari discloses wherein the registration transformation aligns extrinsic or intrinsic parameters of the imaging device with the first image and is a geometric transformation or a planar homography transformation (see Javan Roshtkhari paras. 0019, 0025, and 0028, where a “planar homography transformation” is estimated; and “[t]he module 10 generates a set of one or more output parameters 22, which can include camera parameters, or parameters of the geometric and non-linear transformation, which are numerical values of the intrinsic and extrinsic camera parameters or a subset of them”).
Regarding claim 15, Javan Roshtkhari discloses wherein the registration transformation is used to calibrate the imaging device, and a geometric transformation of the registration transformation represents intrinsic and extrinsic parameters of the imaging device, and a non-linear transformation of the registration transformation comprises optical distortion parameters of the imaging device (see Javan Roshtkhari paras. 0019, 0025, and 0028, where “[t]he module 10 generates a set of one or more output parameters 22, which can include camera parameters, or parameters of the geometric and non-linear transformation, which are numerical values of the intrinsic and extrinsic camera parameters or a subset of them” and “. . . the homography transformation can be augmented with non-linear transformations to model and measure the distortion coefficients in the intrinsic camera parameters, which can be a straightforward process to those familiar with prior camera calibration attempts”).
Regarding claim 16, Javan Roshtkhari discloses wherein the first image shows a part of a sports field and the second image comprises the shape of the sports field (see Javan Roshtkhari Figs. 3-5, where there are images of a sports field and a template of the shape of the sports field).
Regarding claim 17, Javan Roshtkhari discloses wherein the registration transformation comprises a homography transformation between the first image of the sports field and the second image (see Javan Roshtkhari Figs. 3-5, and para. 0019, where a “planar homography transformation” is estimated).
Regarding claim 18, Javan Roshtkhari discloses wherein the second image is adjusted to account for dimensions of the sport field observed in the first image (see Javan Roshtkhari Figs. 3-5, and para. 0027, where “[t]he reference object 14 (i.e. image, or template) of the sport field model is warped according to this initial estimate”).
Regarding claim 19, Javan Roshtkhari discloses wherein the imaging device comprises a broadcast camera and the first image is obtained from a sporting event (see Javan Roshtkhari Figs. 3-5, and para. 0028, where “[a]n exemplary embodiment described illustrates how the proposed camera parameter/pose estimation and image registration can be applied for broadcast sports videos”), and wherein the registration transformation maps each pixel in the first image to its corresponding location in the second image, wherein the second image includes real world coordinates (see Javan Roshtkhari Figs. 3-5, and para. 0024, where “. . . the reference object 14 can be either an image, similar to the input image, or a template of the real world and having a known 3D geometry”).
Regarding claim 20, Javan Roshtkhari discloses further comprising: obtaining a third image from the imaging device; applying the registration parameter estimator to the third image or the adjusted second image, or both the third image and the adjusted second image by mapping pixel data of the adjusted second image to the third image to generate a further estimation of the parametric registration transformation; and updating the one or more parameters of the parametric registration transformation based on the further estimation; and providing output data comprising the updated one or more parameters of the parametric registration transformation (see Javan Roshtkhari para. 0019, where “[t]he following relates to self-camera calibration, planar homography transformation estimation, image registration, and camera pose estimation, which optimizes a learned alignment error objective from observed images, and particularly to continuously re-calibrate and estimate camera parameters from a sequence of observed images” and “[t]he system adaptively adjusts camera parameters, given a new observed image, to minimize the disparity between the re-projected image into a world coordinates system and a world template with known geometric properties” showing that additional images are captured, registered, and used to update the camera parameters).
Regarding claim 24, Javan Roshtkhari discloses wherein the second image is either: a reference image or a template, where the reference image or the template comprises a partial or full representation of contents of the first image; or generated from the first image (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009 and 0052, where the mapping is to “templates”).
Regarding claim 25, Javan Roshtkhari discloses wherein the first and second modalities are different modalities (see Javan Roshtkhari Figs. 1, 4(a), and 4(b), and paras. 0009, 0022, and 0052, where images and templates being registered and/or “registering images of different image modalities” is disclosed).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 5-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Javan Roshtkhari as applied to claim 1 above, and in further view of Ty Nguyen et al: "Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model", ARXIV. ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 September 2017 (2017-09-12), XP081318699 (provided by Applicant with the IDS of 7/28/2025 and hereinafter referred to as “Nguyen”).
Regarding claim 5, Javan Roshtkhari does not explicitly disclose wherein the registration parameter estimator is learned using self-learning techniques wherein a set of previously labelled data is not available.
However, Nguyen discloses wherein the registration parameter estimator is learned using self-learning techniques wherein a set of previously labelled data is not available (see Nguyen Fig. 2, and pgs. 3-5, “IV. Unsupervised Deep Homography Model” where the homography transformation is estimated using unsupervised learning without ground truth labels).
It would have been obvious to one of ordinary skill in the art before the effective filing date to add the unsupervised deep homography model of Nguyen to the homography estimation algorithm of Javan Roshtkhari, because doing so would improve the robustness of the overall algorithm by ensuring a homography is estimated even when labeled data is unavailable.
Regarding claim 6, Javan Roshtkhari discloses wherein the self-learning techniques include designating reference points on the second image, and the registration parameter estimator performs the parametric registration transformation using at least the reference points as parameters (see Javan Roshtkhari paras. 0033-0037, where four preconfigured reference points are used to estimate the homography).
Nguyen also discloses wherein the self-learning techniques include designating reference points on the second image (see Nguyen pgs. 3 and 4, “A. Model Inputs” where the four corners of the image are used as reference points), and the registration parameter estimator performs the parametric registration transformation using at least the reference points as parameters (see Nguyen Fig. 2, and pgs. 3-5, “IV. Unsupervised Deep Homography Model” where the homography transformation is estimated using the reference points).
Regarding claim 7, Javan Roshtkhari discloses wherein the reference points include at least four points and are randomly chosen or preconfigured (see Javan Roshtkhari paras. 0033-0037, where four preconfigured reference points are used to estimate the homography).
Nguyen also discloses wherein the reference points include at least four points and are randomly chosen or preconfigured (see Nguyen pgs. 3 and 4, “A. Model Inputs” where the four corners of the image are used as reference points).
Regarding claim 8, Javan Roshtkhari does not explicitly disclose wherein the at least four points are four corners of a rectangle centered at a center of the first image.
However, Nguyen discloses wherein the at least four points are four corners of a rectangle centered at a center of the first image (see Nguyen pgs. 3 and 4, “A. Model Inputs” where the four corners of the image are used as reference points).
It would have been obvious to one of ordinary skill in the art before the effective filing date to simply substitute the reference point configuration of Javan Roshtkhari with the reference point configuration of Nguyen, because it is predictable that either configuration would succeed at homography estimation, and it is predictable that a centered reference point configuration would improve homography estimation for scenarios when the sports field encompasses the entire image.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Javan Roshtkhari in view of Nguyen as applied to claim 6 above, and in further view of Nie et al., US 2022/0084222 A1 (hereinafter referred to as “Nie”).
Regarding claim 9, Javan Roshtkhari discloses wherein the registration parameter estimator is learned (see Javan Roshtkhari Fig. 3, and paras. 0007, 0027-0029, and 0042-0048, where the mapping functions and estimator 18 are learned).
Javan Roshtkhari does not explicitly disclose using either labeled data or a combination of labeled and unlabeled data.
However, Nie discloses wherein the registration parameter estimator is learned using either labeled data or a combination of labeled and unlabeled data (see Nie Fig. 7, and paras. 0024 and 0039-0046, where both labeled and unlabeled content is used to train the neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date to use the neural network training technique of Nie to train the neural networks of Javan Roshtkhari, as modified by Nguyen, because doing so would improve the robustness and accuracy of the homography estimation by permitting both labeled and unlabeled data and the inclusion of supervision would ensure the resulting homographies are sufficiently accurate for the user supervising the training.
Regarding claim 10, Javan Roshtkhari discloses wherein the registration parameter estimator is adjusted, using supervised machine learning techniques (see Javan Roshtkhari Fig. 3, and paras. 0007, 0027-0029, and 0042-0048, where the mapping functions and estimator 18 are learned and adjusted using neural networks and an initial ground-truth homography).
Javan Roshtkhari does not explicitly disclose using at least one labeled data point.
However, Nie discloses wherein the registration parameter estimator is adjusted using at least one labeled data point, using supervised machine learning techniques (see Nie Fig. 7, and paras. 0024 and 0039-0046, where both labeled and unlabeled content is used to train the neural network, thereby involving supervision and “changes in the homography over time”).
Conclusion
Pertinent prior art: Previously cited Carr et al., US 2013/0176392 A1 (hereinafter referred to as “Carr”) discloses generating a second image from the first image, the second image in a second modality (see Carr Fig. 1, and paras. 0012 and 0075, where “. . . the edge image extracted from camera image 100 . . .”).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW M MOYER whose telephone number is (571)272-9523. The examiner can normally be reached Monday-Friday 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW M MOYER/ Supervisory Patent Examiner, Art Unit 2675
Prosecution Insights