DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is a response to an application filed on 11/24/2025, in which claims 1-20 are pending and ready for examination.
Response to Amendment
Claims 1-5, 7-8, 10-14, and 18-20 are currently amended.
Response to Argument
Applicant's arguments filed 11/24/2025 have been fully considered but they are not persuasive.
With respect to claims rejected under 35 USC 102, the Applicant argues that, see Pg. 11, 14 of filed Remarks, Hinz does not teach “an encoder configured to receive the relevance information and the original image as input, and to generate a latent representation which represents the text caption information and the image feature using the relevance information and the original image” by asserting that Hinz fails to teach a single element which receives an image along with the text embedding as input and output a latent representation, i.e. the image encoder of Hinz receives an image prompt as input, but does not receive the text embedding.
Examiner cannot concur. As taught in at least Para. [0056-57, 59-61], Hinz teaches a single element of an image generation apparatus performs different operations and effectively functions as a text adaptation processor and encoder such that the image generation apparatus, as a text adaptation processor, generates joint embedding information/relevance information indicating a relevance between an image embedding/feature and a text embedding/feature in accordance with text prompt/caption corresponding to an original image prompt. The image generation apparatus, as an encoder, receives joint embedding information/relevance information to generate output image/latent representation representing text prompt/caption information and image prompt/feature in accordance with joint embedding information and input image.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-9, 12-17 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Hinz (US Pub. 20240320872 A1).
Regarding claim 1, Hinz discloses an image encoding apparatus comprising: (Hinz; Fig 2, 4. Para. [0056-57]. An image encoding/generating system, including a processor, is used to encode images.):
a text adaptation processor configured to generate relevance information indicating a relevance between an image feature and a text feature based on text caption information corresponding to an original image (Hinz; Fig. 2, Para. [0056-57, 59-61]. A text encoder/adaptation module is used to generate text embedding/relevance information between an image feature and a text feature in accordance with text description information. A single element of an image generation apparatus performs different operations and effectively functions as a text adaptation processor and encoder such that the image generation apparatus, as a text adaptation processor, generates joint embedding information/relevance information indicating a relevance between an image embedding/feature and a text embedding/feature in accordance with text prompt/caption corresponding to an original image prompt.); and
an encoder configured to receive the relevant information and the original image as input, and to generate a latent representation which represents the text caption information and the image feature using the relevance information and the original image (Hinz; Para. [0058-61]. The image generation apparatus, as an encoder, receives joint embedding information/relevance information to generate output image/latent representation representing text prompt/caption information and image prompt/feature in accordance with joint embedding information and input image. An image generating/encoding module is used to generate an image/representation associated with text description information and image features in accordance with embedding/relevant information and an original image.),
wherein the text adaptation processor is further configured to communicate with the encoder while generating the relevance information (Hinz; Fig. 4, Para. [0057-58, 61]. A text encoder/adaptation module is used to communicate with an image generating/encoding module while generating embedding/relevance information.).
Regarding claim 2, Hinz discloses wherein the text adaptation processor is further configured to generate an embedding vector based on the text caption information, wherein the embedding vector is included in a latent space shared by image and text (Hinz; Para. [0057-58]. A text encoding/adaptation module includes a first encoder generating an embedding vector in accordance with text description information, wherein embedding vector is included in a latent space shared by image and text, also see Para. [0042, 45].); and
generate the relevance information based on the obtained embedding vector and an intermediate image feature generated by the encoder (Hinz; Para. [0110-111]. A second module/GAN is used to generate relevant information based on embedding vector and an intermediate image feature generated by an encoder, also see Para. [0044-45].).
Regarding claim 3, Hinz discloses wherein the text adaptation processor comprises one or more layers configured to obtain the relevance information by gradually reducing a domain difference between the image and the text of the embedding vector using cross-attention processing (Hinz; Para. [0173-176]. A second module/GAN is used to obtain relevance information by reducing difference between an image and a text of embedding vector using cross-attention processing.).
Regarding claim 4, Hinz discloses wherein the text adaptation processor further comprises: a first adaptation layer configured to receive a first intermediate image feature generated by the encoder, and to generate first relevance information based on the first intermediate image feature using the cross-attention processing (Hinz; Para. [0174-177]. A second module/GAN includes multiple layers using at least a first layer used to receive a first image feature generated by an encoder to generate a first relevant information in accordance with a first image feature and a cross-attention processing.); and
a second adaptation layer configured to receive a second intermediate image feature generated by the encoder based on the first relevance information, and to generate second relevance information based on the second intermediate image feature using the cross-attention processing (Hinz; Para. [0174-177]. A second module/GAN is used to receive a second image feature generated by an encoder in accordance with a first relevant information to generate a second relevance information in accordance with a second image feature using cross-attention processing.).
Regarding claim 5, Hinz discloses wherein the text adaptation processor further comprises a third adaptation layer configured to generate an updated text feature based on the second intermediate image feature using the cross-attention processing (Hinz; Para. [0174-177]. A second module includes multiple layers using at least a third layer used to generate an updated text feature in accordance with a second image feature using a cross-attention processing.), and
wherein the second adaptation layer is configured to output the second relevance information based on the updated text feature and the second intermediate image feature (Hinz; Para. [0174-177]. At least a second layer of the multiple layers is used to generate second relevance information in accordance with updated text feature and second image feature.).
Regarding claim 6, Hinz discloses wherein at least one of the first adaptation layer, the second adaptation layer, and the third adaptation layer comprises a linear module configured to apply a linear function to a result of the cross-attention processing (Hinz; Para. [0174-177]. At least a first layer, a second layer, and a third layer of multiple layers include linear module/projection to apply a linear function for cross-attention processing, also see Para. [0144].).
Regarding claim 7, Hinz discloses wherein the encoder comprises: a first encoding layer configured to receive the image and to generate the first intermediate image feature (Hinz; Para. [0097]. An encoding module includes at least a first layer of multiple layers to receive images to generate a first image feature.); and
a second encoding layer configured to generate the second intermediate image feature based on the first intermediate image feature and the first relevance information (Hinz; Para. [0097]. A second layer of multiple layers is used to generate a second image feature in accordance with a first image feature and a first relevance information.).
Regarding claim 8, Hinz discloses wherein the encoder further comprises a third encoding layer configured to output the latent representation based on the second intermediate image feature and the second relevance information (Hinz; An encoding module includes at least a third layer of multiple layers to generate a representation in accordance with a second image feature and a second relevance information.).
Regarding claim 9, Hinz discloses wherein each of the first encoding layer, the second encoding layer, and the third encoding layer comprises at least one of a convolutional neural network (CNN), a residual block, and an attention module (Hinz; Para. [0194-196]. Each of a first layer, a second layer, and a third layer includes at least a CNN, residual blocks, and cross-attention processing.).
Claims 12-17 are directed to an image encoding method for encoding an image using an image encoding apparatus, the method comprising sequence of processing steps corresponding to the same as claimed in claims 1-2, 4-5, 7-8, and are rejected for the same reason of anticipation as outlined above.
Claim 11 is rejected under 35 U.S.C. 102(a)(2) as being anticipated by Johnston (US Pat. 11869221 B2).
Regarding claim 11, Johnston discloses an image decoding apparatus comprising: at least one processor; and a memory configured to store instructions which, when executed by the at least one processor, cause the image decoding apparatus to: perform entropy decoding on a bitstream generated by an image encoding apparatus based on a latent representation which represents text caption information and an image feature associated with an original image (Johnston; Col. 19, Ln. 24-58. An image decoding system, including at least a processor, is used to perform entropy coding on a bitstream generated by an encoder in accordance with a latent data of text information and image feature of an original image, also see Col. 18, Ln. 28-30.); and
generate a reconstructed image corresponding to the original image based on a result of the entropy decoding (Johnston; Col. 19, Ln. 24-58. A decoding module is used to generate a reconstructed image for an original image in accordance with an entropy decoding.),
wherein the latent representation is generated by providing the original image and relevance information indicating a relevance between the image feature and a text feature corresponding to the text caption information as input to an encoder (Hinz; Para. [0058-61]. The image generation apparatus, as an encoder, receives joint embedding information/relevance information to generate output image/latent representation representing text prompt/caption information and image prompt/feature in accordance with joint embedding information and input image. An image generating/encoding module is used to generate an image/representation associated with text description information and image features in accordance with embedding/relevant information and an original image.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 10 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hinz (US Pub. 20240320872 A1) in view of Johnston (US Pat. 11869221 B2).
Regarding claim 10, Hinz disclose an image encoding apparats (Hinz; See remarks regarding claim 1 above.).
But it does not specifically disclose an entropy encoder configured to perform entropy encoding on the latent representation to transform the latent representation into a bitstream.
However, Johnston teaches an entropy module configured to perform entropy encoding on the latent representation to transform the latent representation into a bitstream (Col. 19, Ln. 40-45. An entropy encoding is used to transform latent information into bitstream.).
Therefore, it would have been obvious to a person with ordinary skill in the pertinent before the effective filing date of the claimed invention to modify the video coding system of Hinz to adapt an image encoding approach, by incorporating Johnston’s teaching wherein entropy encoding is used to code latent representation, for the motivation to perform entropy encoding on latent data (Johnston; Abstract.).
Claim 18 is directed to an image encoding method for encoding an image using an image encoding apparatus, the method comprising sequence of processing steps corresponding to the same as claimed in claim 10, and is non-patentable over the prior art for the same reason as previously indicated.
Regarding claim 19, Hinz discloses an electronic device comprising: at least one processor; and a memory configured to store one or more instructions which, when executed by the at least one processor, cause the electronic device to (Hinz; Para. [0056]. An electronic system includes a memory storing instructions and a processor.)
generate, using a text adaptation processor, relevance information indicating a relevance between an image feature and a text feature based on text caption information corresponding to an original image (Hinz; Fig. 2, Para. [0057]. A text encoder/adaptation module is used to generate text embedding/relevance information between an image feature and a text feature in accordance with text description information.); and
provide the relevance information and the original image as input to an encoder, and generate, using the encoder, a latent representation which represents the text caption information and the image feature using the relevance information and the original image (Hinz; Para. [0058, 61]. An image generating/encoding module is used to generate an image/representation associated with text description information and image features in accordance with embedding/relevant information and an original image.),
wherein the text adaptation processor is configured to communicate with the encoder while generating the relevance information (Hinz; Fig. 4, Para. [0057-58, 61]. A text encoder/adaptation module is used to communicate with an image generating/encoding module while generating embedding/relevance information.).
But it does not specifically disclose perform entropy encoding on the latent representation to generate a bitstream, and to perform entropy decoding on the generated bitstream to generate a reconstructed bitstream; and generate, using a decoder, a reconstructed image corresponding to the original image based on a result of the entropy decoding.
However, Johnston teaches perform entropy encoding on the latent representation to generate a bitstream, and to perform entropy decoding on the generated bitstream to generate a reconstructed bitstream (Johnston; Col. 19, Ln. 24-58. An image decoding system, including at least a processor, is used to perform entropy coding on latent data of text information and image feature of an original image and to perform entropy decoding to generate reconstructed bitstream, also see Col. 18, Ln. 28-30.); and
generate, using a decoder, a reconstructed image corresponding to the original image based on a result of the entropy decoding (Johnston; Col. 19, Ln. 24-58. A decoding module is used to generate a reconstructed image for an original image based on entropy decoding.).
Therefore, it would have been obvious to a person with ordinary skill in the pertinent before the effective filing date of the claimed invention to modify the video coding system of Hinz to adapt an image encoding approach, by incorporating Johnston’s teaching wherein entropy encoding is used to code latent representation, for the motivation to perform entropy encoding on latent data (Johnston; Abstract.).
Regarding claim 20, modified Hinz further teaches wherein the at least one processor is further configured to train at least one of the text adaptation processor, the encoder, and the decoder such that a value of a predetermined multi-modal objective function is minimized (Hinz; Para. [0075-76, 80]. A processor is used to implement a training module to train text module, encoding module, decoding module for minimizing a function.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Antsfeld (US Pub. 20250259321 A1) teaches a video coding system that performs training of models for visual odometry.
Park (US Pat. 12524937 B2) teaches a video coding system that performs text-based image generation.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALBERT KIR whose telephone number is (571)272-6245. The examiner can normally be reached Monday - Friday, 8:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jay Patel can be reached at (571) 272-2988. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALBERT KIR/ Primary Examiner, Art Unit 2485