DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-11 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-4 and 8-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kapoor et al (EP4120136A1) in view of Yusiong et al (multi-scale autoencoders, 2019).
Regarding claims 1, 10 and 11, Kapoor teaches a computer implemented method for evaluating a digital image, comprising:
providing the digital image;
(Kapoor, "The present invention is further directed to a detection system and/or a detection method, in particular for detecting an attack, using a VQ-VAE. In particular, by means of that, a trivial detection system using a VQVAE can be designed such that the difference of denoised/reconstructed image with the input image can be compared to a threshold.", [0105]; a computer-implemented method for detecting/evaluating an attack on an input digital image using a VQ-VAE system; "During inference, the input image is directly fed to the VQ-VAE", [0139]; providing a digital input image to the system)
providing a first part of a predetermined model, wherein the predetermined model is configured for determining a semantic segmentation of the digital image with a second part of the predetermined model,
(Kapoor, "In particular, autoencoder are and/or are based on neural networks which consist of an encoder and a decoder.", [0045]; Yusiong; "we introduce a novel encoder-decoder network architecture for semantic image segmentation of outdoor scenes called SAsiANet.", [abstract]; "The decoder section of SAsiANet does not merely perform upsampling; it also performs "autoencoding" before each upsampling step to exploit features at multiple scales and obtain better segmentation outputs", p591; Kapoor teaches a predetermined model with a first part (encoder) and a second part (decoder), but Kapoor relies on a separate downstream network to perform semantic segmentation; Yusiong teaches an encoder-decoder autoencoder model where the second part (decoder) is explicitly configured to determine the semantic segmentation directly)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Yusiong into the system or method of Kapoor in order to integrate the semantic segmentation task directly into the decoder, and to directly evaluate the latent-space quantization error for anomaly detection as established by Kapoor's underlying principles. The combination of Kapoor and Yusiong also teaches other enhanced capabilities.
(In the arguments filed on 5/29/2026 regarding the motivation to combine Kapoor and Yusiong, applicant argues that modifying Kapoor’s decoder to output a semantic segmentation map per Yusiong would destroy Kapoor’s anomaly detection framework because a pixel-level RGB reconstruction error could no longer be calculated. The examiner respectfully disagrees. A person having ordinary skill in the art would recognize that Kapoor’s detection framework is fundamentally rooted in the latent space. Kapoor explicitly states that its detection "relies on the assumption that the clean images are usually closer to the codebook vector representation than attacked images" ([0105]). Because this latent-space distance (quantization error) is already explicitly computed by Kapoor during the quantization step (see Eq. 2, [0140]), it would have been obvious to a person having ordinary skill in the art to evaluate this quantization error directly to detect anomalies. Doing so preserves Kapoor’s core anomaly detection mechanism while adopting Yusiong's end-to-end segmentation decoder, achieving Yusiong's benefit of "high accuracy pixel-wise prediction" (Yusiong, Abstract) and improving computational efficiency by eliminating the need for full RGB image reconstruction. A proposed combination is not improper simply because it modifies one specific downstream calculation (reconstruction error), so long as the underlying principle of the invention (detecting attacks via codebook vector distances) is maintained and improved)
The combination of Kapoor and Yusiong further teaches:
wherein the first part is configured to determine a feature depending on the digital image, and wherein the second part is configured to determine the semantic segmentation depending on the feature;
(Kapoor, "the encoder neural network encodes the sensor data by generating an encoder output, in particular by mapping the sensor data from the input space representation into the (low dimensional) latent space representation of the sensor data.", [0047]; "the target perception unit (40) being supplied with the target input sensor data as input is configured to process the target input sensor data by means of an artificial neural network.", [claim 6]; the first part (encoder) determines a feature (latent space representation) from the image, and the second part (target perception unit) determines semantic segmentation depending on the reconstructed data from the feature; Yusiong, "encoder-decoder networks which have an encoder module that systematically reduces the spatial maps to obtain higher contextual representations and a decoder module that gradually recovers the spatial information", p589; "involves exploiting features across multiple scales when upsampling the output of the encoder to obtain better spatial and contextual information", [abstract]; a first part (encoder) producing “higher contextual representations” (features) from the image, and a second part (decoder) using the encoder output to recover spatial information and produce segmentation outputs)
determining the feature depending on the digital image with the first part;
(Kapoor, "the encoder performs a non-linear mapping of the input image x to a bottleneck vector", [0140]; the first part (encoder) determines the feature (bottleneck vector) from the digital image)
providing a set of quantizations for quantizing the feature;
(Kapoor, "a discrete set of (codebook) vectors", [0049]; "This vector is then quantized (denoted as Q(.)) to prototype vectors in the codebook", [0140]; providing a codebook, which serves as the set of quantizations for the feature)
determining a quantization of the feature depending on the set of quantizations and depending on the feature;
(Kapoor, "This vector is then quantized... to prototype vectors in the codebook... using a nearest neighbor search as follows: ... argmin_j ||f_e(x) - e_j||, eq. (2)", [0140]; "The objective of training is to learn the weights... using the following loss function... ||f_e(x) - sg[e]||_2^2, eq. (3)", [0141]; Kapoor explicitly teaches calculating the distance (error) between the continuous encoder feature f_e(x) and the nearest discrete codebook quantization vector e_j. The system determines this distance both to perform the nearest neighbor search (eq. 2) and to calculate the commitment loss representing the quantization error (eq. 3). Therefore, Kapoor determines a quantization error depending on the feature and the quantization)
evaluating the digital image depending on the quantization error.
(Kapoor, "If this is high, it is classified as adversarial, if not it would be clean. This relies on the assumption that the clean images are usually closer to the codebook vector representation than attacked images", [0105]; The claim requires evaluating the digital image depending on the quantization error, not solely on it. As noted in the motivation statement, Kapoor establishes that attacked images exhibit a larger distance between their latent features and the codebook vectors (i.e., a larger quantization error). By integrating Yusiong's segmentation decoder, it would be obvious to apply Kapoor's stated assumption directly by evaluating the explicitly calculated latent quantization error to classify the image as clean or adversarial. Furthermore, even if evaluated downstream as originally implemented in Kapoor, the final reconstruction error mathematically depends on the quantization error generated in the latent space. Thus, the evaluation of the digital image depends on the quantization error)
Regarding claim 2, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising:
providing the second part of the model, wherein the second part is configured for determining the semantic segmentation of the digital image depending on the quantization; and
(Kapoor, Fig. 2; "The decoder of the VQVAE takes the quantized vector ... as input", [0150]; providing a second part (the decoder) that takes the quantized vector as input; Yusiong, Fig. 1; "The decoder section of SAsiANet does not merely perform upsampling; it also performs "autoencoding" before each upsampling step to exploit features at multiple scales and obtain better segmentation outputs", p591; a second part (decoder section) is configured for determining the semantic segmentation; as established in Claim 1, combining these teaches a second part configured to determine the semantic segmentation depending on the quantization; incorporate Kapoor and Yusiong to configure the decoder of the autoencoder to directly output a semantic segmentation map using the quantized latent features as its input)
determining the semantic segmentation of the digital image depending on the quantization with the second part.
(Kapoor, Fig. 2; "The decoder of the VQVAE takes the quantized vector ... as input", [0150]; using the quantized feature as input to the second part (decoder); Yusiong, Fig. 1; "The decoder section of SAsiANet does not merely perform upsampling; it also performs "autoencoding" before each upsampling step to exploit features at multiple scales and obtain better segmentation outputs", p591; determining the semantic segmentation with the second part; the combination teaches determining the semantic segmentation depending on the quantization)
Regarding claim 3, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 2, wherein the providing of the set of quantizations includes
providing a reference for the semantic segmentation of the digital image,
(Yusiong, "The framework requires an RGB image and its corresponding ground truth during training.", p590; providing a ground truth, which serves as the reference for the semantic segmentation; incorporating Yusiong into Kapoor would provide a ground truth reference to train the network for accurate pixel-wise semantic classification rather than mere image reconstruction)
determining the semantic segmentation of the digital image, and
(Yusiong, "The decoder section of SAsiANet does not merely perform upsampling; it also performs "autoencoding" before each upsampling step to exploit features at multiple scales and obtain better segmentation outputs", p591; determining the predicted semantic segmentation output)
determining a quantization in the set of quantizations depending on a difference between the reference and the semantic segmentation and depending on the quantization error.
(Yusiong, loss function eq. (1), p590; determining the difference between the reference (ground truth ytrue) and the semantic segmentation (ypred) via a cross-entropy loss function; Kapoor, "gradient of the reconstruction error is then back-propagated through the decoder and to the encoder using a straight-through gradient estimator. The objective of training is to learn the weights of the encoder, decoder as well as the codebook vectors using the following loss function", [0141], eq. (3); learning/determining the set of quantizations (codebook vectors) by back-propagating the task loss along with the quantization error ||sg[f_e(x)] - e||_2^2; when Kapoor and Yusiong are combined, the codebook vectors are learned depending on the semantic segmentation loss (difference between reference and segmentation) and the quantization error; incorporating Yusiong into Kapoor would update and optimize the codebook vectors using gradients derived from the semantic segmentation cross-entropy loss combined with the quantization error, thereby directly tailoring the discrete latent representations for scene parsing)
Regarding claim 4, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the providing of the predetermined model includes training the first part to determine the feature and the second part to determine the semantic segmentation depending on the feature.
(Kapoor, "The objective of training is to learn the weights of the encoder, decoder as well as the codebook vectors", [0141]; training the first part (encoder) and second part (decoder); Yusiong, "The solution involves training SAsiANet to classify each pixel in the image correctly.", p590; training the model (including the second part) to classify pixels to determine the semantic segmentation; the combination teaches training the first part to determine the feature and the second part to determine the semantic segmentation; Yusiong into Kapoor would jointly train the encoder and decoder weights specifically for the task of pixel-wise semantic image segmentation)
Regarding claim 8, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the providing of the digital image includes capturing the digital image with a camera of an at least partially autonomous vehicle or an automated optical inspection device.
(Kapoor, "a (digital) image captured by a camera ", [0066]; "at least partially autonomously controlled, vehicle", [0120]; providing the digital image by capturing it with a camera of an at least partially autonomous vehicle)
Regarding claim 9, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the evaluating of the digital image includes detecting an anomaly when the quantization error exceeds a threshold or not detecting the anomaly otherwise.(Kapoor, "a trivial detection system using a VQVAE can be designed such that the difference of denoised/reconstructed image with the input image can be compared to a threshold. If this is high, it is classified as adversarial, if not it would be clean. This relies on the assumption that the clean images are usually closer to the codebook vector representation than attacked images", [0105]; " Preferably, the detection system and/or the detection method comprise a threshold for the reconstruction error of the trained VQ-VAE ... the reconstruction error of the trained VQ-VAE is determined and/or calculated (via computer-implemented method steps) and an attack parameter is determined based on the reconstruction error (via a computer-implemented method step)", [0106]; threshold-based anomaly detection classifies the image as adversarial (anomaly detected) when the error exceeds a threshold and as clean otherwise; Kapoor grounds this mechanism in the assumption that clean images sit closer to their codebook vector representations, i.e., that the quantization error is lower for clean images; although Kapoor's operational threshold comparison is stated in terms of reconstruction error, the underlying detection rationale is directly tied to the quantization error, consistent with the parent claim mapping)
Claim(s) 5 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kapoor et al (EP4120136A1) in view of Yusiong et al (multi-scale autoencoders, 2019) and further in view of Liu et al (Hyperspherical Quantization, 2022).
Regarding claim 5, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination does not expressly disclose but Liu teaches the method according to claim 1, further comprising:
(Liu, “Our proposed method includes the preprocessing and quantization steps”, sec. 4, p3:c2)
determining the feature with a predetermined normalization; and(Liu, “The input vector 𝐱 and each column vector 𝐰j ∈ ℝ^n of 𝐖 satisfy ‖𝐰j‖2=1,‖𝐱‖2=1 for all j=1,…,m”, sec. 3.1, p2:c2; etermining a feature, such as a full-precision input vector or weight vector, with a predetermined normalization where the L2 norm is equal to 1)
determining the quantization for the feature with the predetermined normalization.
(Liu, “3.2. Ternary Quantizer ... eq. (2)”, p2:c2; “Based on Eq. (2) and hyperspherical learning, we have ‖𝐰^j‖2=1 and ‖𝐰j‖2=1”, sec. 3.4, p3:c1; determining the quantization for the feature, represented by the ternary weight vector wj^, with the exact same predetermined normalization where the L2 norm is equal to 1; motivation of combining Liu with Kapoor and Yusiong: see comments on claim 7; limitations in claims 5 and 7 are highly correlated)
Regarding claim 7, the combination of Kapoor and Yusiong teaches its/their respective base claim(s).
The combination of Kapoor, Yusiong and Liu teaches the method according to claim 1,
wherein the feature is a vector and the quantization of the feature is a vector,
(Liu,
PNG
media_image1.png
154
390
media_image1.png
Greyscale
both the feature wj and the quantization of the feature w^j are vector)
wherein determining the quantization error includes determining a cosine distance between the feature and the quantization of the feature.
(Liu, "The cosine distance between full-precision and ternary weights is: ... eq. (8)", p3:c1; "reduces the cosine distance between the full-precision and ternary weights, thus reducing the bias of the straight-through gradient estimator during ternary quantization.", [abstract]; determining the quantization error or bias by determining and minimizing a cosine distance between the full-precision feature vector and its quantized ternary counterpart) It would have been obvious to a person of ordinary skill in the art to modify the base framework by incorporating Liu into Kapoor and Yusiong in order to reduce the quantization error and bias between full-precision features and their quantized counterparts; by applying Liu's predetermined hyperspherical normalization and cosine distance metric to the quantization process taught in Kapoor and Yusiong, one would achieve the predictable benefit of minimizing the discrepancy of the quantized vectors, thereby yielding a more stable, highly compressed, and accurate machine learning model for environment perception)
Allowable Subject Matter
Claim(s) 6 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 6 recite(s) limitation(s) related to upscaling the feature to determine quantization, then downscaling to calculate the error. There are no explicit teachings to the above limitation(s) found in the prior art cited in this office action and from the prior art search.
Response to Arguments
Applicant's arguments filed on 5/29/2026 with respect to one or more of the pending claims have been fully considered but they are not persuasive.
Regarding claim(s) 1, Applicant, in the remarks, argues that the combination of the cited references fails to teach limitations “determining a quantization error depending on the feature and the quantization; and evaluating the digital image depending on the quantization error” as recited in claim 1.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments on claim 1 for details.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 7/25/2026
Prosecution Insights