DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention. Claim 8 recites the limitation “wherein feature maps outputted by intermediate layers of the teacher and the student neural networks are chosen the intermediate work products of the teach and student neural networks, respectively”. It is not clear what is being chosen and for what purpose. The sentence appears to be grammatically incorrect. Clarification is requested.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 9-15 and 16-17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by D1.1
With regard to claim 1, D1 teach method for training a student neural network to adopt the behavior of a teacher neural network that is trained to perform a given processing on input images (see abstract: knowledge distillation from teacher to student network), comprising the following steps: providing a set of training images (see fig. 3: training data sets); producing, from at least one training image from the set of training images, one or more style-augmented versions that: have the same semantic content as the training image but differ from the training image in their style (see abstract, § 2.3, § 3.1 step 7: data augmentation where data has different style but same semantic content); processing the training image and the style- augmented versions, by the teacher neural network on the one hand, and by the student neural network on the other hand (see fig. 2, 3: teacher and student network trained on training data including augmented data); evaluating, using a predetermined loss function, to which extent outputs and/or intermediate work products produced by the student neural network from the one training image and the style-augmented versions are in agreement with outputs and/or intermediate work products produced by the teacher neural network from the same training image and style-augmented versions (see fig. 2, 3: loss function to evaluate divergence between teacher and student network trained on training data and augmented training data); and optimizing parameters that characterize behavior of the student neural network towards a goal of improving a value of the loss function (see fig. 2, 3, § 3.2: parameters optimized to improve loss function).
With regard to claim 2, D1 teach method of claim 1, wherein the producing of each style-augmented version of the training image includes: extracting, from the training image, a semantic content, obtaining, based at least in part on a style source image, a style; and processing, by a trained generative model, the semantic content and the style into the style-augmented version (see fig. 1, abstract, § 3.1 step 7, § 2.3: style augmented training data is generated by keeping the semantic content by changing the style form a different source).
With regard to claim 3, D1 teach method of claim 2, wherein: (i) the extracting of the semantic content from the training image, and/or (ii) the obtaining of the style based on the style source image, is performed by feeding the training image and the style source image, into a trained feature extractor network that is configured to determine both a semantic content and a style from an input image (see fig. 1, § 2.3, abstract: training data is augmented based on feeding different style source image and training image into a trained network, augmented image has same semantic content but different style).
With regard to claim 4, D1 teach method of claim 2, wherein the style is chosen to be an interpolation between a first style of the training image and a second style of the style source image (see § 2.3: interpolation).
With regard to claim 5, D1 teach method of claim 2, wherein the style source image differs from one or more training images in at least one aspect that is distinct from the semantic content (see abstract, § 3.1 step 7, § 2.3: style source image has different style content than the training image).
With regard to claim 6, D1 teach method of claim 5, wherein the aspect in which the style source image differs from all training images includes one or more of: a time of day at which the style source image was acquired; a season of the year in which the style source image was acquired; weather and/or lighting conditions under which the style source image was acquired; imperfections and/or disturbances in the style source image; and a camera setup with which the style source image was acquired (see abstract, § 3.1 step 7, § 2.3: style source image inherently acquired at a different time or inherently has different imperfections or disturbances or noise).
With regard to claim 9, D1 teach method of claim 1, wherein the outputs of the teacher and the student neural networks are chosen to be logits, and/or other unaggregated results outputted by the teacher and the student neural networks, respectively (see fig. 2, § 3.2: logits).
With regard to claim 10, D1 teach method of claim 1, wherein the loss function measures a distance between outputs and/or intermediate work products produced by the teacher neural network on the one hand, and by the student neural network on the other hand (see fig. 3, abstract, § 2.3, § 3.2: loss measures the difference or divergence between teacher and student network).
With regard to claim 11, D1 teach method of claim 10, wherein the Kullback-Leibler divergence is chosen as a measure for the distance (see abstract: KL or Kullback-Leibler measure).
With regard to claim 12, D1 teach method of claim 1, wherein: the student neural network is configured to produce outputs with respect to a given task, the training images are labelled with ground truth with respect to the given task; and the loss function further measures a difference and/or distance between the outputs produced from each training image and the ground truth for the respective training image (see fig 3, abstract, § 1 ¶ 1, § 3.2 steps 1-3: loss function measures the difference or divergence between student and teacher performing a task such as classification or recognition, training data is labeled).
With regard to claim 13, D1 teach method of claim 12, wherein the loss function further measures a difference and/or distance between the outputs produced from each style-augmented image version and a ground truth for a training image to which the style-augmented version relates (see fig. 3, § 3.2 steps 1-3: loss function measures difference or divergence between outputs of teacher and student network).
With regard to claim 14, D1 teach method of claim 1, wherein the student neural network is configured as an image classifier, and/or a semantic segmentation model, and/or an object detector (see abstract, § 1 ¶ 1, fig. 3: image or spectrogram classification).
With regard to claim 16, see discussion of claim 1.
With regard to claim 17, see discussion of claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over D1.
With regard to claim 7, D1 teach method of claim 2, but fails to explicitly teach wherein at least one augmented version of a training image, and/or at least one style source image, is obtained from a trained generative diffusion model using a text prompt that is indicative of a desired style of the style-augmented version. However, Examiner takes Official Notice that generative diffusion models are extremely well known in the art before the effective filing date. Examiner also takes Official Notice to the fact that user input to select a style or other modifications is extremely well known in the art before the effective filing date. One skilled in the art would have found it obvious to incorporate known teachings of using generative diffusion model with user input to select style into the configuration of D1 yielding predictable and enhanced results, particularly generative diffusion models offer superior training stability and output diversity.
With regard to claim 15, D1 teach method of claim 1, further comprising: providing input images that have been acquired using at least one sensor to the trained student neural network (see abstract, fig,. 4: spectrogram images are inherently acquired by sensors). D1 fails to explicitly teach determining an actuation signal from an output of the trained student neural network based on the provided input images; and actuating, using the actuation signal, a vehicle, a driving assistance system, and/or a robot, and/or a surveillance system, and/or a quality inspection system, and/or a medical imaging system. However, Examiner takes Official Notice to the fact that it is extremely well known in the art to control vehicle or robot or driving assistance based on results of image classification or recognition and it would have been obvious to incorporate known teachings into the configuration of D1 yielding predictable results. In particular, the results of the classification or recognition performed on the spectrogram images maybe used as input to control some device.
Pertinent Art
Wang et al.2 teach is related to collaborative distillation for ultra-resolution universal style transfer.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AVINASH YENTRAPATI whose telephone number is (571)270-7982. The examiner can normally be reached on 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AVINASH YENTRAPATI/Primary Examiner, Art Unit 2672
1 Tripathi, Achyut Mani, and Konark Paul. "Data augmentation guided knowledge distillation for environmental sound classification." Neurocomputing 489 (2022): 59-77.
2 Wang, Huan, et al. "Collaborative distillation for ultra-resolution universal style transfer." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
Prosecution Insights