DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4 and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “the image encoder is additionally fine-tuned so that an image similar to an actual image is output when the actual image is input” in claims 4 and 10 is a relative term which renders the claim indefinite. The term “similar” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
The specification teaches using actual images as both input and output during fine-tuning and optimizing the encoder while the decoder is fixed. See instant specification ¶¶ [0052]-[0055] and [0072]-[00074]. However, the specification does not identify an objective boundary for determining when the generated output image is sufficiently similar to the actual input image, such as a reconstruction loss threshold, pixel error threshold, feature distance criterion, SSIM/PSNR criterion, or other measurable standard. Nor does the claim itself identify the required degree of similarity. Accordingly, one of ordinary skill in the art would not be informed with reasonable certainty of the metes and bounds of the requirement that an image similar to an actual image be output, and claims 4 and 10 are indefinite. See MPEP § 2173.05(b). For purposes of prior-art examination, the phrase has been interpreted as requiring reconstruction of an output image corresponding to the actual input image, consistent with the disclosed autoencoder fine-tuning process.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
virtual image generation unit, image regeneration unit, object measurement unit, object measurement module in claims 1 and 6; and their dependent claims.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4-8, and 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al., US 2018/0181802 A1 (published June 28, 2018) ("Chen") in view of Inoue et al., US 2019/0259175 A1 (published August 22, 2019) ("Inoue").
Regarding claim 1:
Chen disclose: a system for vision measurement of object information based on deep learning (Chen teaches a neural network system for analyzing an input image and determining body shape and body pose. See Chen FIGS. 1-5 and ¶¶ [0065]-[0066], [0080]-[0081]), comprising:
a virtual image generation unit configured to generate individual virtual images using model variable values of an object model that represents a shape and posture of the object (Chen teaches generating synthetic training images using a deformable body model. Parameters defining body pose and body shape are applied to the model to generate the synthetic images, and corresponding body-shape and body-pose data are retained as training information. See Chen ¶¶ [0053], [0055]-[0059], [0062]-[0064], particularly ¶ [0056]);
an image regeneration unit comprising an image encoder, which includes an encoding neural network trained using the model variable values and the virtual images (Chen teaches autoencoder neural network model 500 including encoder 504. Encoder 504 receives a depth map corresponding to a synthetic training image and maps the image to feature vector 506, which includes portions representing body shape parameter 706 and body pose parameter 708. See Chen ¶¶ [0068]-[0070] and FIG. 5 and 7. Chen further trains the network using synthetic image subsets in which pose or shape is varied while other attributes are held fixed, thereby training corresponding feature vector portions to represent pose and shape. See Chen ¶¶ [0074]-[0077]),
and an image decoder, which includes a decoding neural network trained using the model variable values and the virtual images (Chen teaches decoder 508, which receives feature vector 506 containing pose and shape related information and decodes that vector into synthesized depth map 510. The autoencoder is trained to minimize the difference between the input and synthesized depth maps. See Chen ¶¶ [0068]-[0071] and FIG. 5 and 8. Chen further teaches training such that the pose portion and shape portion of the feature vector reproduce the corresponding pose and shape when decoded. See Chen ¶¶ [0074]-[0077]); and
an object measurement unit configured to output a measurement value for the object by using the image encoder (Chen teaches using the trained encoder on an input image to determine body shape and body pose. See Chen ¶¶ [0080]-[0081]);
Chen is silent about the encoder having been additionally fine-tuned using actual images of the object in the image regeneration unit
However, in the same field of endeavor, Inoue teaches: which has been additionally fine-tuned using actual images of the object in the image regeneration unit (Inoue teaches initially training VAE1 using synthetic images and then training corresponding VAE2 using real images, with respective encoders and a common decoder. See Inoue ¶¶ [0033]-[0035], [0043], [0047]-[0049]. More specifically, Inoue teaches copying the synthetic trained parameters to VAE2 and then training on real images while fixing the decoder layers and adapting the encoder parameters. See Inoue ¶ [0072]. Inoue then uses the adapted real image encoder output to determine object pose, including position and orientation values. See Inoue ¶¶ [0050]-[0051], [0072]-[0073]).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Chen's synthetic image trained encoder architecture according by Inoue by further adapting the encoder using actual object images while retaining the representation previously learned using synthetic imagery. Inoue recognizes the synthetic to real domain difference and teaches addressing that difference by adapting the real image encoder while retaining the synthetic trained decoder. See Inoue ¶¶ [0035], [0049], [0072]-[0073]. The modification would predictably improve pose and shape determination from actual images while preserving the representation learned from abundant synthetic training data.
Regarding claim 2:
Chen further teaches: wherein the image encoder is configured such that the encoding neural network is trained to output each of the model variable values corresponding to the virtual images by using the virtual images as input information (Chen teaches encoder 504 receiving a depth map corresponding to a synthetic training image as input and generating feature vector 506. See Chen ¶ [0069]. The feature vector includes components corresponding to body shape parameter 706 and body-pose parameter 708. See Chen ¶ [0070]. Chen further trains the network using synthetic image subsets in which pose or shape is varied while other parameters are fixed, such that the resulting feature-vector values represent the varied pose and shape parameters. See Chen ¶¶ [0074]-[0077]).
Regarding claim 4:
Chen in view Of Inoue teaches: wherein the image regeneration unit is configured such that output information of the image encoder is combined to serve as input information of the image decoder and the image encoder is additionally fine-tuned so that an image similar to an actual image is output when the actual image is input (Chen teaches encoder 504 producing feature vector 506, which decoder 508 receives to generate synthesized depth map 510, and trains the autoencoder to minimize the difference between the input and reconstructed depth maps. See Chen ¶¶ [0069]-[0071]. Inoue teaches adapting the corresponding autoencoder using real images, including adapting the real image encoder while retaining and fixing the decoder. See Inoue ¶¶ [0035], [0049], [0072]-[0073]).
Applying Inoue's real image encoder adaptation to Chen's architecture would therefore result in the encoder being additionally fine-tuned using actual images while its output continues to serve as decoder input for reconstruction of the actual image content. Accordingly, Chen in view of Inoue renders claim 4 obvious.
Regarding claim 5:
Inoue further teaches: wherein image regeneration unit is configured such that decoding layer parameter values constituting the decoding neural network are fixed, while the encoding layer parameter values constituting the encoding neural network are set to change (Inoue teaches during real image training of VAE2, fixing the decoder layers while adapting only the parameters of the encoder receiving the real images. See Inoue ¶ [0072]; see also ¶ [0035]).
Regarding claim 6:
Chen in view Of Inoue teaches: wherein the object measurement unit comprises:
an optimized encoder configured to obtain encoding layer parameter values from the image regeneration unit trained using the actual images of the object (Inoue teaches training VAE2 with real images after initialization from the synthetic trained VAE and adapting the encoder parameters while fixing the decoder. See Inoue ¶ [0072]. The resulting encoder is an optimized real image encoder whose encoding layer parameters are obtained through actual image training); and
an object measurement module configured to output the measurement value for the object using the model variable value corresponding to output information of the optimized encoder (Chen teaches that encoder generated feature information includes body shape and body pose information and uses the trained network to determine body shape and pose from an input image. See Chen ¶¶ [0070], [0074]-[0077], [0080]-[0081]. Inoue further teaches bypassing the decoder and supplying the adapted encoder representation to an MLP for object pose determination. See Inoue ¶¶ [0050]-[0051]).
Regarding claims 7, 8, and 10-12: claim 7 is the method counterpart of claim 1, and claims 8 and 10-12 correspond respectively to system claims 2 and 4-6; therefore, rejected in the same manner as applied above.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Inoue, as applied above, and further in view of Kowalski et al., US 2021/0335029 A1 (published October 28, 2021) ("Kowalski").
Regarding claim 3:
wherein the image decoder is configured such that the decoding neural network is trained to output each of the virtual images corresponding to the model variable values (Chen teaches decoder 508 receiving a feature vector containing pose- and shape-related portions and decoding the feature vector into a synthesized image, with training structured so the respective pose and shape portions reproduce corresponding pose and shape in the output. See Chen ¶¶ [0070]-[0071], [0074]-[0077]);
Chen does not specifically teach: using the model variable values themselves as decoder input information during training.
However, in the same field of endeavor, Kowalski teaches: by using the model variable values as input information (Kowalski teaches training a neural image generation system using synthetic images and the known renderer parameters by which those images were generated, such that known correspondences between renderer parameters and synthetic images train a generative model having the same meaningful input parameterization as the synthetic graphics pipeline. See Kowalski ¶ [0035]. Kowalski further teaches synthetic data supplied as parameter vectors θ describing corresponding synthetic images. See Kowalski ¶ [0038]) semantically meaningful parameters corresponding to inputs of the synthetic rendering apparatus. See ¶¶ [0042]-[0044]; a neural decoder that generates the output image. See ¶ [0046]; and training using an image loss between a synthetic image corresponding to a set of attributes and an output image generated from those attributes. See ¶ [0054]. Kowalski additionally teaches that an object pose parameterization, such as head pose represented by Euler angles or quaternions, may be input directly to decoder G without requiring an encoder. See Kowalski ¶ [0091]; see also ¶ [0133]).
Therefore, It would have been obvious to one of ordinary skill in the art to implement Chen's decoder according to Kowalski such that known pose or shape model parameters associated with the synthetic training images are used as semantically meaningful decoder input information. The modification would provide predictable and independently controllable image generation from the same parameter information used to generate the synthetic training imagery by the use of known input conditioning mechanics applied to standard deep graphics decoding networks.
Regarding claim 9: the claim is the method counterpart of claim 3. Accordingly, claim 9 is rejected for the same reasons set forth with respect to claim 3.
Relevant prior art not relied upon
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Guay (US 20220392099 A1) teaches a system for estimating the pose of articulated objects such as people, animals, or robots. The system first learns from synthetic or manually labeled training images, where the pose labels distinguish the left side and right side of the object. It then improves itself using unlabeled real-world images by reconstructing images from predicted poses. The training engine can obtain captured images (e.g., one or more videos) of the object from data-collection component and fine-tune image encoder, pose estimator, uplift model, and/or image renderer to the appearance of the object and improve the performance of image encoder.
Brumleve (US 20210158109 A1) teaches a generator that is configured to receive an input consisting of the control vector produced by the encoder. The generator is then configured to output a generated image or new image. The generator and encoder are configured to function in tandem but may be trained with varied or differing training image sets, resulting in differing output images from one image generation to another
Svitov (US 20250209712 A1) teaches an autoencoder including an encoder trained to encode the neural texture into a lower dimensional latent representation of the neural texture input to the denoising U-Net architecture, and a decoder trained to decode an output of the denoising U-Net architecture to the inpainted neural texture. Then the pipeline (with added pretrained VQGAN autoencoder) was fine-tuned end-to-end for additional 20,000 steps to reduce the neural renderer θ artifacts when processing neural textures processed by the VQGAN autoencoder.
Goforth (US 20210150228 A1) teaches a computer-vision style system for autonomous vehicles that looks at a partial 3D point cloud of an object and predicts both what the object looks like and how it is oriented in space. The training approach first learns shape completion, then learns pose prediction while freezing some weights, and can later be fine-tuned jointly.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights