Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is a responsive to the application filed on 10/03/2023.
Claims 1-13 are pending.
Claims 1-13 are rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed inventio n and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-13 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (“CMLocate: A cross-modal automatic visual geo-localization framework for a natural environment without GNSS information”, 2023) hereinafter Liu, in view of Cao et al (“Synergistic Self-supervised and Quantization Learning”, 2022) hereinafter Cao.
Regarding claims 1, 7, and 13, Liu teaches a panoramic perception method applied in a computer including a processing circuit and a storage device coupled to the processing circuit, the storage device including a plurality of hardware circuits for storing a source database and a target database, the panoramic perception method comprising; a panoramic perception system comprising: a processing circuit; and a storage device coupled to the processing circuit, the storage device including a plurality of hardware circuits for storing a source database and a target database; wherein the processing circuit performs the following; and a non-transitory computer-readable medium, when read by a computer, the computer executing the panoramic perception method according to Claim 1 (sections 3.3.3 teaches “Our experiments are based on the PyTorch platform, Intel®Xeon®E5-2680 v4 CPU and a GeForce RTX 3080 graphics processing unit (GPU)”):
performing a first pretraining on a plurality of weights of a training model using the source database (sections 3.4.2-3.4.3 teach “We initialize our model with weights pretrained on the ImageNet [48] dataset. The purpose of this is to prevent overfitting when there are fewer query images available. Before the model is trained, the resolution of the input image needs to be adjusted”);
performing a second pretraining with data augmentation on the plurality of weights of the training model using the source database (sections 3.4.2-3.4.3 teach “To train the model, we perform geometric augmentation on both the query images and the database images. This includes random shifting, rotation, and flipping of the images. The augmentation strategies are applied to all images in the triplet” and learning the model weights);
performing a combined training on the plurality of weights of the training model using both the source database and the target database (sections 3.4.2-3.4.3 teach training the model on the dataset, and also “We use weakly supervised triplet ranking loss to train the model…We trained the model for 100 epochs with the complete dataset and 50 epochs with the local dataset”);
performing a quantization-aware training on the plurality of weights of the training model using the source database and the target database (sections 3.4.3-3.5 teaches quantized index (quantization-aware) inference model training steps with the skyline image datasets);
performing a post training quantization on the plurality of weights of the training model using the target database (sections 3.5 and 4.3.1 teach performing model performance reconstruction error with quantization (post training quantization) on the datasets); and
performing panoramic perception by the training model (sections 1 and 3.4.2-3.4.3 teach “LineNet proposed in this paper is used to extract the skyline in the panorama image to obtain a panoramic skyline image (performing panoramic perception by the training model), which is a binary image like the DEM panoramic skyline image”; and training the model on panoramic image data (alternative performing panoramic perception by the training model)).
Liu at least implies performing a quantization-aware training on the plurality of weights of the training model using the source database and the target database (see mappings above); however, Cao teaches performing a quantization-aware training on the plurality of weights of the training model using the source database and the target database (sections 3.2 and 4.1 teach “Our motivation is to train a quantization-friendly pretrained model, hence we proposed to introduce quantization into contrastive learning…We simply adopt the commonly used uniform quantizer for both weights and activations”; wherein “The main experiments are conducted on three benchmark datasets, i.e., CIFAR-10, CIFAR-100 [29] and ImageNet [39]” for model training).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to implement Cao’s teachings of quantization machine learning model training of model weights using data into Liu‘s teaching of pretraining models, quantized index model training, and quantization model performance on panoramic datasets in order to “greatly improves the performance when quantized to lower bits, but also boosts the performance of full precision models” (Cao, sections 3.2, 4.1, and 5).
Regarding claims 2 and 8, the combination of Liu and Cao teach all the claim limitations of claims 1 and 7 above; and further teach wherein the plurality of weights of the training model are randomly generated during initialization (Cao, section 4.3 teaches “Following previous practices, we also finetune the pretrained models with a randomly initialized linear classifier on ImageNet with 1% and 10% labeled data”).
Liu and Cao are combinable for the same rationale as set forth above with respect to claims 1, 7, and 13.
Regarding claims 3 and 9, the combination of Liu and Cao teach all the claim limitations of claims 1 and 7 above; and further teach wherein data in the source database and the target database are 32-bit floating-point data (Cao, sections 2 and 3.2 teach “Quantization is a method that converts the weights and activations in networks from full precision (i.e., 32-bit floating-point) to fixed-point integers”; and “Notice that the quantized network and the floating-point network share weights, hence when we backprop on the quantized network Wq using STE, the gradients will directly operate on the floating-point network W”).
Liu and Cao are combinable for the same rationale as set forth above with respect to claims 1, 7, and 13.
Regarding claims 4 and 10, the combination of Liu and Cao teach all the claim limitations of claims 1 and 7 above; and further teach wherein data augmentation includes mosaic data augmentation, Gaussian blur, contrast adjustment, saturation adjustment, hue adjustment, crop or rotation (Liu, section 3.4.2 teaches “To train the model, we perform geometric augmentation on both the query images and the database images. This includes random shifting, rotation, and flipping of the images. The augmentation strategies are applied to all images in the triplet”).
Regarding claims 5 and 11, the combination of Liu and Cao teach all the claim limitations of claims 1 and 7 above; and further teach wherein the quantization-aware training comprises: inputting a quantized input into a first target operator, wherein the quantized input is a quantized output from a previous layer or a quantized data from the source database (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach a “backbone” ResNet model being trained with input quantization data; wherein “The main experiments are conducted on three benchmark datasets, i.e., CIFAR-10, CIFAR-100 [29] and ImageNet [39]” for model training);
extracting a plurality of weights from a layer of the training model (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach a “backbone” ResNet model being trained with input quantization data for determining model weight values);
determining a first quantization scale (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach determining quantization parameters including “scale”);
quantizing the weights with the first quantization scale to generate a plurality of quantized weights (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach quantizing the backbone model with model weights with the determined scale parameter);
performing operations on the quantized input and the quantized weights in the layer of the training model to obtain an output (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach fine-tuning the model to obtain result values, and “based on the fine-tuned FP model, we conduct either PTQ or QAT to evaluate the performance after quantization”); and
quantizing the output with the first quantization scale to obtain a quantized output (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach “based on the fine-tuned FP model, we conduct either PTQ or QAT to evaluate the performance after quantization”, being post training quantization).
Liu and Cao are combinable for the same rationale as set forth above with respect to claims 1, 7, and 13.
Regarding claims 6 and 12, the combination of Liu and Cao teach all the claim limitations of claims 1 and 7 above; and further teach wherein the post training quantization comprises:
inputting a pre-quantized input into a second target operator to obtain a first feature (Cao, sections 3.1-3.2, 4.1-4.2, and Fig. 3 teach multiple “backbone” ResNet models being trained with input quantization data; wherein “The main experiments are conducted on three benchmark datasets, i.e., CIFAR-10, CIFAR-100 [29] and ImageNet [39]” for model training);
inputting the pre-quantized input into a first quantization scale to obtain a quantized input (Cao, sections 3.1-3.2, 4.1-4.2, and Fig. 3 teach quantizing the given backbone model with model weights at a determined scale parameter to obtain an output);
inputting the quantized input into a first target operator to obtain a second feature, wherein the second target operator has a precision higher than the first target operator (Cao, sections 3.1-3.2, 4.1-4.2, and Fig. 3 teach quantizing the given backbone model with model weights at a higher determined scale parameter to obtain an output of different bit-widths);
determining a second quantization scale (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach determining the quantization scaling parameter according to different “bit-widths”, being “q”); and
scaling-shifting the second feature into a third feature using the second quantization scale and comparing the third feature with the first feature to determine whether to change the second quantization scale (Cao, sections 3.1-3.2, 4.1, and Fig. 3 teach “based on the fine-tuned FP model, we conduct either PTQ or QAT to evaluate the performance after quantization”, being post training quantization of iterative training and changing the quantization scaling parameter according to different “bit-widths”, being “q”).
Liu and Cao are combinable for the same rationale as set forth above with respect to claims 1, 7, and 13.
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hatamizadeh et al (US Pub 20240185034) teach pre-training of machine learning models, augmentation of image data for training, and quantization of model weights.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLINT MULLINAX whose telephone number is 571-272-3241. The examiner can normally be reached on Mon - Fri 8:00-4:30 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.M./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights