DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/19/2024 is/are being considered by the examiner.
Election/Restrictions
Claims 7, 9, 10, 17, 19, and 20 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected species, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 01/14/2026.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-6 and 8 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Tripathy et al, “Learning image-to-image translation using paired and unpaired training samples” (published at https://arxiv.org/pdf/1805.03189, May 2018).
Regarding claim 1, Tripathy teaches a computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a source training dataset sampled from a source domain, the source training dataset comprising a plurality of source training images (see Tripathy Figure 2, “phase-2” for unpaired data, wherein “x” samples are from the source domain and section 2, referring to “Image-to-image translation using unpaired training data”);
obtaining a target training dataset sampled from a target domain, the target training dataset comprising a plurality of target training images (see Figure 2, wherein “y” samples are from the target domain);
for each respective source training image in the source training dataset, translating, using a forward generator neural network, the respective source training image to a respective translated target image in the target domain (see Figure 2, sample xi being fed to G1 and section 3.2, “The generator G1 provides a mapping from domain X to domain Y denoted as G1 : X → Y (e.g. semantic map to photograph)”);
receiving, as input to a target discriminator network, a random one of the plurality of target training images or the translated target images (see Figure 2, output G1(xi) or “y” being fed to D1 and caption which indicates D1 is a discriminator);
predicting, using the target discriminator network, that the random one of the plurality of target training images or the translated target images belongs to the target domain or was translated from the source domain to the target domain;
determining an adversarial loss term based on the prediction of the target discriminator network, the adversarial loss term representing how accurately the forward generator neural network generates an output that can be considered as being from the target domain (see section 3.2, “Generator networks are trained using adversarial loss that is implemented by two discriminator networks D1 and D2. D1 aims to distinguish between transformation results G1(x) and given training samples y ∈ Y”); and
jointly training the forward generator neural network and the target discriminator network based on the adversarial loss term (see section 2, first paragraph, “The generator network produces output samples which should ideally be indistinguishable from the training distribution. Since it is infeasible to engineer a proper loss function, the assessment is done by the discriminator network. Both networks are differentiable, which allows to train them jointly” and equation (1) which shows a joint loss function for G1 and D1).
Regarding claim 2, Tripathy teaches all the limitations of claim 1, and further teaches wherein the operations further comprise, for each respective target training image in the target training dataset, translating, using a backward generator neural network, the respective target training image to a respective translated source image in the source domain (see Tripathy Figure 2, sample yj being fed into G2 in the “backward cycle” of phase-2 for unpaired data and section 3.2, “whereas the second generator learns the inverse of this mapping denoted as G2 : Y → X (e.g. photograph to semantic map)”).
Regarding claim 3, Tripathy teaches all the limitations of claim 2, and further teaches wherein the operations further comprise:
receiving, as input to a source discriminator network, a random one of the plurality of source training images or the translated source images (see Tripathy Figure 2, output G2(yi) or “x” being fed to D2 and caption which indicates D2 is a discriminator); and
predicting, using the source discriminator network, that the random one of the plurality of source training images or the translated source images belongs to the source domain or was translated from the target domain to the source domain (see Tripathy section 3.2, “D1 aims to distinguish between transformation
results G1(x) and given training samples y ∈ Y. D2 works similarly for G2(y)
and x ∈ X”).
Regarding claim 4, Tripathy teaches all the limitations of claim 3, and further teaches wherein the operations further comprise:
determining a second adversarial loss term based on the prediction of the source discriminator network, the second adversarial loss term representing how accurately the backward generator neural network generates an output that can be considered as being from the source domain (see Tripathy section 3.2, “Generator networks are trained using adversarial loss that is implemented by two discriminator networks D1 and D2”); and
jointly training the backward generator neural network and the source discriminator network based on the second adversarial loss term (see Tripathy section 2, first paragraph, “The generator network produces output samples which should ideally be indistinguishable from the training distribution. Since it is infeasible to engineer a proper loss function, the assessment is done by the discriminator network. Both networks are differentiable, which allows to train them jointly” and section 3.2, “The corresponding objective LGAN(G2(Y ),D2(X)) for G2 and D2 is obtained
similarly”).
Regarding claim 5, Tripathy teaches all the limitations of claim 2, and further teaches wherein the operations further comprise determining a circularity loss component that ensures the forward generator neural network and the backward generator neural network are a pair of inverse translation operations (see Tripathy section 3.3 “Following [4], we circumvent the problem by adding a cycle consistency loss that encourages the cascaded mapping to reproduce the original input. Mathematically, G2(G1(x)) ≈ x and G1(G2(y)) ≈ y, where x ∈ X and y ∈ Y . The intuition behind the cycle consistency is that if we translate from one domain to another and then back again we should arrive where we started (e.g. translating a sentence from one language to another and then back again)”).
Regarding claim 6, Tripathy teaches all the limitations of claim 5, and further teaches wherein the circularity loss component further ensures:
a source training image from the source training dataset can be translated to a circularly-translated source image that is the same as the source training image after the source training image is translated by the forward generator neural network and subsequently by the backward generator neural network; and
a target training image from the target training dataset can be translated to a circularly-translated target image that is the same as the target training image after the target training image is translated by the backward generator neural network and subsequently the forward generator neural network (see Tripathy Figure 2, xr and yr output from G2 and G1 in phase-2 being fed along with xi and yi into D3 and D4, respectively, and section 3.3, second paragraph).
Regarding claim 8, Tripathy teaches all the limitations of claim 1, and further teaches wherein:
the plurality of source training images each comprise a map of a geographic region; and the plurality of target training images each comprise an aerial photo of a corresponding geographic region (see Tripathy Figure 4, second row with caption and section 4.2).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11-16 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tripathy et al, “Learning image-to-image translation using paired and unpaired training samples” (published at https://arxiv.org/pdf/1805.03189, May 2018) in view of Kim et al, U.S. Publication No. 2019/0042882.
Regarding claim 11, Tripathy teaches a system configured to perform operations comprising:
obtaining a source training dataset sampled from a source domain, the source training dataset comprising a plurality of source training images (see Tripathy Figure 2, “phase-2” for unpaired data, wherein “x” samples are from the source domain and section 2, referring to “Image-to-image translation using unpaired training data”);
obtaining a target training dataset sampled from a target domain, the target training dataset comprising a plurality of target training images (see Figure 2, wherein “y” samples are from the target domain);
for each respective source training image in the source training dataset, translating, using a forward generator neural network, the respective source training image to a respective translated target image in the target domain (see Figure 2, sample xi being fed to G1 and section 3.2, “The generator G1 provides a mapping from domain X to domain Y denoted as G1 : X → Y (e.g. semantic map to photograph)”);
receiving, as input to a target discriminator network, a random one of the plurality of target training images or the translated target images (see Figure 2, output G1(xi) or “y” being fed to D1);
predicting, using the target discriminator network, that the random one of the plurality of target training images or the translated target images belongs to the target domain or was translated from the source domain to the target domain;
determining an adversarial loss term based on the prediction of the target discriminator network, the adversarial loss term representing how accurately the forward generator neural network generates an output that can be considered as being from the target domain (see section 3.2, “Generator networks are trained using adversarial loss that is implemented by two discriminator networks D1 and D2. D1 aims to distinguish between transformation results G1(x) and given training samples y ∈ Y”); and
jointly training the forward generator neural network and the target discriminator network based on the adversarial loss term (see section 2, first paragraph, “The generator network produces output samples which should ideally be indistinguishable from the training distribution. Since it is infeasible to engineer a proper loss function, the assessment is done by the discriminator network. Both networks are differentiable, which allows to train them jointly” and equation (1) which shows a joint loss function for G1 and D1).
Tripathy does not expressively teach
data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations.
However, Kim in as similar invention in the same field of endeavor teaches a system configured to perform operations comprising:
obtaining a source training dataset sampled from a source domain, the source training dataset comprising a plurality of source training images;
obtaining a target training dataset sampled from a target domain, the target training dataset comprising a plurality of target training images (see Kim Figure 2, training data 250 and paragraph [0031], wherein “reference images” are the target training dataset);
for each respective source training image in the source training dataset, translating, using a forward generator neural network, the respective source training image to a respective translated target image in the target domain (see Figure 2, translator 210 with generator 213 and paragraphs [0033]-[0035]);
receiving, as input to a target discriminator network (see Figure 2, discriminator 220), a target training image or a translated target image associated with the target training image (see Figure 3 which shows the training algorithm associated with Figure 2, Xsrc being transformed by “generator” based on Xref and Xtrans along with Xref being fed to “discriminator”. Paragraph [0056] indicates that either Xref or Xtrans ultimately analyzed by the discriminator);
predicting, using the target discriminator network, that the target training image or translated target image associated with the target training image belongs to the target domain or was translated from the source domain to the target domain (see paragraph [0056]);
determining an adversarial loss term based on the prediction of the target discriminator network, the adversarial loss term representing how accurately the forward generator neural network generates an output that can be considered as being from the target domain; and jointly training the forward generator neural network and the target discriminator network based on the adversarial loss term (see paragraph [0038]) as taught in Tripathy, the system comprising
data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations (see paragraph [0062]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of data processing hardware and memory hardware for performing operations as taught in Kim with the system taught in Tripathy, the motivation being to automate the process thereby making it easier to replicate and be used in different systems.
Regarding claim 12, Tripathy in view of Kim teaches all the limitations of claim 11, and further teaches wherein the operations further comprise, for each respective target training image in the target training dataset, translating, using a backward generator neural network, the respective target training image to a respective translated source image in the source domain (see Tripathy Figure 2, sample yj being fed into G2 in the “backward cycle” of phase-2 for unpaired data and section 3.2, “whereas the second generator learns the inverse of this mapping denoted as G2 : Y → X (e.g. photograph to semantic map)”).
Regarding claim 13, Tripathy in view of Kim teaches all the limitations of claim 12, and further teaches wherein the operations further comprise:
receiving, as input to a source discriminator network, a random one of the plurality of source training images or the translated source images (see Tripathy Figure 2, output G2(yi) or “x” being fed to D2 and caption which indicates D2 is a discriminator); and
predicting, using the source discriminator network, that the random one of the plurality of source training images or the translated source images belongs to the source domain or was translated from the target domain to the source domain (see Tripathy section 3.2, “D1 aims to distinguish between transformation
results G1(x) and given training samples y ∈ Y. D2 works similarly for G2(y)
and x ∈ X”).
Regarding claim 14, Tripathy teaches all the limitations of claim 13, and further teaches wherein the operations further comprise:
determining a second adversarial loss term based on the prediction of the source discriminator network, the second adversarial loss term representing how accurately the backward generator neural network generates an output that can be considered as being from the source domain (see Tripathy section 3.2, “Generator networks are trained using adversarial loss that is implemented by two discriminator networks D1 and D2”); and
jointly training the backward generator neural network and the source discriminator network based on the second adversarial loss term (see Tripathy section 2, first paragraph, “The generator network produces output samples which should ideally be indistinguishable from the training distribution. Since it is infeasible to engineer a proper loss function, the assessment is done by the discriminator network. Both networks are differentiable, which allows to train them jointly” and section 3.2, “The corresponding objective LGAN(G2(Y ),D2(X)) for G2 and D2 is obtained
similarly”).
Regarding claim 15, Tripathy teaches all the limitations of claim 12, and further teaches wherein the operations further comprise determining a circularity loss component that ensures the forward generator neural network and the backward generator neural network are a pair of inverse translation operations (see Tripathy section 3.3 “Following [4], we circumvent the problem by adding a cycle consistency loss that encourages the cascaded mapping to reproduce the original input. Mathematically, G2(G1(x)) ≈ x and G1(G2(y)) ≈ y, where x ∈ X and y ∈ Y . The intuition behind the cycle consistency is that if we translate from one domain to another and then back again we should arrive where we started (e.g. translating a sentence from one language to another and then back again)”).
Regarding claim 16, Tripathy teaches all the limitations of claim 15, and further teaches wherein the circularity loss component further ensures:
a source training image from the source training dataset can be translated to a circularly-translated source image that is the same as the source training image after the source training image is translated by the forward generator neural network and subsequently by the backward generator neural network; and
a target training image from the target training dataset can be translated to a circularly-translated target image that is the same as the target training image after the target training image is translated by the backward generator neural network and subsequently the forward generator neural network (see Tripathy Figure 2, xr and yr output from G2 and G1 in phase-2 being fed along with xi and yi into D3 and D4, respectively, and section 3.3, second paragraph).
Regarding claim 18, Tripathy teaches all the limitations of claim 11, and further teaches wherein:
the plurality of source training images each comprise a map of a geographic region; and the plurality of target training images each comprise an aerial photo of a corresponding geographic region (see Tripathy Figure 4, second row with caption and section 4.2).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635