DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/17/2025 has/have been considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 22-23, 25-26 and 28 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12175723 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 22-23, 25-26 and 28 of instant application 18945375 can be anticipated by claim1 of U.S. Patent No. 12175723 B2.
Instant Application 18945375
U.S. Patent No. 12175723 B2
22. (New) A computer-implemented method for generating a refined image, comprising: obtaining a source image; generating a source feature representation of the source image; obtaining a plurality of source keypoint locations in the source image; obtaining a plurality of target keypoint locations; generating a transported feature representation by modifying the source feature representation based on the source keypoint locations and the target keypoint locations; and generating a refined image from the transported feature representation.
23. The method of claim 22, wherein generating the refined image comprises processing the transported feature representation using a refinement neural network to generate the refined image.
25. The method of claim 22, wherein the source respective source feature representation comprises a source feature map that includes respective source feature vectors for each of a plurality of locations.
26. The method of claim 22, wherein the refined image is a modified version of the source image, the modifying being based on the source keypoint locations and the target keypoint locations.
28. The method of claim 22, wherein the refined image approximates an image of a same environment as the source image captured at a different time.
1. A method of training a keypoint extraction machine learning model having a plurality of keypoint model parameters, wherein the keypoint extraction machine learning model is configured to receive an input image and to process the input image in accordance with the keypoint model parameters to generate a plurality of keypoint locations in the input image, the method comprising: obtaining a source image of an environment; obtaining a target image of the environment that was captured at a different time than the source image; generating a reconstruction of the target image, the generating comprising: processing the source image using a feature extraction neural network having a plurality of feature extraction network parameters and in accordance with current values of the feature extraction network parameters to generate a source feature map that includes respective source feature vectors for each of a plurality of locations; processing the source image using the keypoint extraction machine learning model in accordance with current values of the keypoint model parameters to generate a plurality of source keypoint locations, wherein each source keypoint location is a respective one of the plurality of locations; processing the target image using the feature extraction neural network in accordance with the current values of the feature extraction network parameters to generate a target feature map that includes respective target feature vectors for each of the plurality of locations; processing the target image using the keypoint extraction machine learning model in accordance with current values of the keypoint model parameters to generate a plurality of target keypoint locations, wherein each target keypoint location is a respective one of the plurality of locations; generating, from the source feature map, a transported feature map by augmenting the source feature map with data from the target feature vectors for the target keypoint locations; generating the reconstruction of the target image from the transported feature map, comprising processing the transported feature map using a refinement neural network to generate the reconstruction; and determining an update to the current values of the keypoint model parameters by determining gradients with respect to the keypoint model parameters of an objective function that measures an error between the target image and the reconstruction of the target image.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 2-5, 7-8, 12-14, 16-17, 22-23 and 25-28 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jakab et al (arXiv:1806.07823v2 13 Dec 2018), hereinafter Jakab.
-Regarding claim 2, Jakab discloses a method performed by one or more computers (one or more computers has to be used in order to implement the method shown in Jakab’s FIG. 1), the method comprising (Abstract; FIGS. 1-14
PNG
media_image1.png
345
780
media_image1.png
Greyscale
):obtaining a source image of an environment (FIG.1, image
x
); obtaining a target image of the environment (FIG.1, image
x
'
); generating a reconstruction of the target image (FIG. 1, caption, image
x
^
'
), the generating comprising (FIG. 1; Sec. 3.): processing the source image using a feature extraction neural network having a plurality of feature extraction network parameters and in accordance with current values of the feature extraction network parameters to generate a source feature map that includes respective source feature vectors for each of a plurality of locations (FIG. 1, network Φ (up branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 4.1., 1st paragraph, “The landmark detector … It is composed of sequential blocks consisting of two convolutional layers each …”); obtaining a plurality of source keypoint locations (FIG. 1; Page 3, 2nd paragraph, “
PNG
media_image2.png
65
770
media_image2.png
Greyscale
”); obtaining a target feature map that includes respective target feature vectors for each of the plurality of locations; obtaining a plurality of target keypoint locations (FIG. 1, network Φ (bottom branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 3.1.; Sec. 4.1., 1st paragraph); generating, from the source feature map, a transported feature map by augmenting the source feature map with data from the target feature vectors for the target keypoint locations (FIG. 1, caption, “2D Gaussians (
y
'
) are rendered from these keypoints and stacked along with the image features extracted from
x
, to reconstruct the target …); generating the reconstruction of the target image from the transported feature map (FIG. 1, up branch, left), comprising processing the transported feature map using a refinement neural network to generate the reconstruction (FIG. 1, up branch, generator network Ψ and decoder (network on the left side); Sec. 3.2); and training the refinement neural network on an objective function that measures an error between the target image and the reconstruction of the target image (Sec. 3.2, 1st paragraph, “… the generator network is optimized to minimize a reconstruction error
L
(
x
'
,
x
^
'
) …”, 2nd paragraph).
-Regarding claim 12, Jakab discloses a method performed by one or more computers (one or more computers has to be used in order to implement the method shown in Jakab’s FIG. 1), the method comprising (Abstract; FIGS. 1-14):obtaining a source image of an environment (FIG.1, image
x
); obtaining a target image of the environment (FIG.1, image
x
'
); generating a reconstruction of the target image (FIG. 1, caption, image
x
^
'
), the generating comprising (FIG. 1; Sec. 3.): processing the source image using a feature extraction neural network having a plurality of feature extraction network parameters and in accordance with current values of the feature extraction network parameters to generate a source feature map that includes respective source feature vectors for each of a plurality of locations (FIG. 1, network Φ (up branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 4.1., 1st paragraph, “The landmark detector … It is composed of sequential blocks consisting of two convolutional layers each …”); obtaining a plurality of source keypoint locations (FIG. 1; Page 3, 2nd paragraph, “
PNG
media_image2.png
65
770
media_image2.png
Greyscale
”); obtaining a plurality of target keypoint locations (FIG. 1, network Φ (bottom branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 3.1.; Sec. 4.1., 1st paragraph); generating, from the source feature map, a transported feature map by augmenting the source feature map based on the target keypoint locations (FIG. 1, caption, “2D Gaussians (
y
'
) are rendered from these keypoints and stacked along with the image features extracted from
x
, to reconstruct the target …); generating the reconstruction of the target image from the transported feature map (FIG. 1, up branch, left), comprising processing the transported feature map using a refinement neural network to generate the reconstruction (FIG. 1, up branch, generator network Ψ and decoder (network on the left side); Sec. 3.2); and training the refinement neural network on an objective function that measures an error between the target image and the reconstruction of the target image (Sec. 3.2, 1st paragraph, “… the generator network is optimized to minimize a reconstruction error
L
(
x
'
,
x
^
'
) …”, 2nd paragraph).
-Regarding claim 22, Jakab discloses a computer-implemented method for generating a refined image (one or more computers has to be used in order to implement the method shown in Jakab’s FIG. 1), comprising (Abstract; FIGS. 1-14):obtaining a source image (FIG.1, image
x
); obtaining a source feature representation of the source image FIG. 1, network Φ (up branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 4.1., 1st paragraph, “The landmark detector … It is composed of sequential blocks consisting of two convolutional layers each …”); obtaining a plurality of source keypoint locations in the source image (FIG. 1; Page 3, 2nd paragraph, “
PNG
media_image2.png
65
770
media_image2.png
Greyscale
”); obtaining a plurality of target keypoint locations (FIG. 1, network Φ (bottom branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 3.1.; Sec. 4.1., 1st paragraph); generating a transported feature representation by modifying the source feature representation based on the source keypoint locations and the target keypoint locations (FIG. 1, caption, “2D Gaussians (
y
'
) are rendered from these keypoints and stacked along with the image features extracted from
x
, to reconstruct the target …); and generating a refined image from the transported feature representation (FIG. 1, up branch, generator network Ψ and decoder (network on the left side); Sec. 3.2).
-Regarding claim 3, Jakab discloses the method of claim 2. Jakab further discloses wherein obtaining a target feature map that includes respective target feature vectors for each of the plurality of locations comprises: processing the target image using the feature extraction neural network in accordance with the current values of the feature extraction network parameters to generate the target feature map (FIG. 1, network Φ (bottom branch), caption; Sec. 3., Page 3, 2nd – 3rd paragraphs; Sec. 4.1., 1st paragraph, “The landmark detector … It is composed of sequential blocks consisting of two convolutional layers each …”).
-Regarding claims 4 and 13, Jakab discloses the method of claim 2 and the method of claim 12. Jakab further discloses wherein the refinement neural network is a neural network that maps the feature map to an image having the same resolution as the source and target images (FIG. 1, image
x
^
'
,
x
,
x
'
).
-Regarding claims 5 and 14, Jakab discloses the method of claim 2 and the method of claim 12. Jakab further discloses wherein obtaining the target image comprises: randomly sampling a time delta from a set of possible time deltas; and selecting, as the target image, the image that was captured at the sampled time delta from the source image (Page 8, Sec. 4.3., 1st paragraph, “Image pairs are extracted by randomly sampling frames from the same video sequence, with the additional constraint of maintaining the time difference within the range 3-30 frames …”).
-Regarding claims 7 and 16, Jakab discloses the method of claim 2 and the method of claim 12. Jakab further discloses determining an update to the current values of the feature extraction network parameters by determining gradients with respect to the feature extraction network parameters of the objective function that measures the error between the target image and the reconstruction of the target image (Page 6, 1st paragraph; Page 4, Sec. 3.2.; Note: this is a basic step for any learning algorithm).
-Regarding claims 8 and 17, Jakab discloses the method of claim 2 and the method of claim 12. Jakab further discloses wherein the feature extraction neural network is a convolutional neural network that maps an image having the resolution of the source and target images to a feature map that includes respective vectors for each of the plurality of locations (FIG. 1; Page 3, 2nd paragraph, “
PNG
media_image2.png
65
770
media_image2.png
Greyscale
”; Sec. 4.1., 1st paragraph, “The landmark detector … It is composed of sequential blocks consisting of two convolutional layers each …”).
-Regarding claim 23, Jakab discloses the method of claim 22. Jakab further discloses wherein generating the refined image comprises processing the transported feature representation using a refinement neural network to generate the refined image (FIG. 1).
-Regarding claim 25, Jakab discloses the method of claim 22. Jakab further discloses wherein the source respective source feature representation comprises a source feature map that includes respective source feature vectors for each of a plurality of locations (FIG. 1; Page 3, 2nd -3rd paragraphs, Sec. 3.1.; Sec. 4.1., 1st paragraph).
-Regarding claim 26, Jakab discloses the method of claim 22. Jakab further discloses wherein the refined image is a modified version of the source image, the modifying being based on the source keypoint locations and the target keypoint locations (FIG. 1).
-Regarding claim 27, Jakab discloses the method of claim 26. Jakab further discloses wherein the modifying results in object features, corresponding to the source keypoint locations in the source image, being aligned with the target keypoint locations in the refined image (FIGS. 1, 4-5; Page 8, Sec. 4.3., 2nd paragraph, “For each annotated keypoint, a maximally matching unsupervised keypoint is identified by solving bipartite linear assignment using mean distance as the cost. Regressed keypoints consistently track the annotated points …”).
-Regarding claim 28, Jakab discloses the method of claim 22. Jakab further discloses wherein the refined image approximates an image of a same environment as the source image captured at a different time (FIG. 1; Page 1, 2nd paragraph; Page 8, Sec. 4.3., 1st paragraph).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jakab et al (arXiv:1806.07823v2 13 Dec 2018), hereinafter Jakab in view of Wang et al (US 20190122329 A1), hereinafter Wang.
-Regarding claims 6 and 15, Jakab discloses the method of claim 2 and the method of claim 12.
Jakab does not disclose wherein the objective function is an error between the reconstruction and the target image in a pixel space of the reconstruction and the target image.
In the same field of endeavor, Wang teaches a method image reconstruction using a facial landmark determination model having a cascade multichannel convolutional neural network (CMC-CNN) to process both the target and the source images (Wang: FIGS 1-9). Wang further teaches wherein the objective function is an error between the reconstruction and the target image in a pixel space of the reconstruction and the target image (Wang: [0031]-[0032]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Jakab with the teaching of Wang by using an error between the reconstruction and the target image in a pixel space of the reconstruction and the target image for objective function in order to achieve the similarity between the two images.
Allowable Subject Matter
Claim 9-11, 18-21 and 24 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and overcome claim rejections in above section of “Double Patenting”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAO LIU whose telephone number is (571)272-4539. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAO LIU/Primary Examiner, Art Unit 2664
Prosecution Insights