DETAILED ACTION
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 18-20 are rejected under 35 U.S.C. 101 because they are directed to non-statutory subject matter "computer readable medium" and “computer program product”. The broadest reasonable interpretation of a claim drawn to a computer readable medium covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media. Transitory signal does not fall within a statutory category since it is clearly not a series of steps or acts to constitute a process, not a mechanical device or combination of mechanical devices to constitute a machine, not a tangible physical article or object which is some form of matter to be a product and constitute a manufacture, and not a composition of two or more substances to constitute a composition of matter.
Note that a claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation "non-transitory" to the claim.
Allowable Subject Matter
Claims 2-8, and 11-13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable Hatamizadeh et al. (US 2023/0394781 hereinafter “Hat”) in view of YUAN et al. (Pub. No. US 2023/0162481) further in view of Chang et al. (Pub. No. US 2017/0046601).
Regarding claims 1, 14, 16, and 17, Hat teaches an image classification method, performed by a computer device [Para. 27 “the method 100 may be performed by a device comprised of a processing unit, a program, custom circuitry, or a combination thereof.”, the method comprising: obtaining a sample image (image to-be-classified) (input image), and mapping a plurality of image patches in the sample image using an image classification model (vision transformer), to obtain a feature map of the sample image [Para. 28, “In operation 102, an input image is processed through at least one stage of a vision transformer to obtain feature representations for the input image”; para. 50 “The stem layer 202 obtains image patches for the image and projects those image patches into an embedding space having a defined dimension.”], the feature map comprising feature patches [Para. 58 “As shown, local self-attention is computed on feature patches within the same local window only”], and each of the feature patches being obtained through feature mapping on each of the plurality of image patches [Para. 50 “The stem layer 202 obtains image patches for the image and projects those image patches into an embedding space having a defined dimension”; Para. 58 “As shown, local self-attention is computed on feature patches within the same local window only”. A sample image could also be considered an image to-be-classified]; performing combined processing (processing stages) of at least one layer (stage) on the feature map (feature representation) using the image classification model (vision transformer), to obtain a combined-processed feature map (feature representations) outputted through the combined processing (series of stages) of the at least one layer (stage) [Para. 30 “As mentioned above, the input image is processed through at least one stage of the vision transformer”; Para. 51 “The projected image patches are output from the stem layer 202 and processed through a series of stages 304A-D of the vision transformer 300.” and Para. 37 “In operation 104, the feature representations are output.”] wherein in combined processing of each layer: determining a self-attention window (local window), a window size (local window size) of the self-attention window (local window) matching a size of a feature patch (feature patches) in a target feature map (feature map) inputted at the layer [Para. 29 “The input image is apportioned into a plurality of local windows”; Para. 58 “As shown, local self-attention is computed on feature patches within the same local window only”; Para. 77 “In an embodiment, the features are processed for dimension matching to a local window size”; extracting an intermediate feature (local feature) of a covered feature patch (feature patches within the same local window) according to the self-attention window (local window), the covered feature patch (feature Paches within the same local window) being a feature patch covered by the self-attention window (local window) in the target feature map (feature representation) [Para. 31 “With respect to the present embodiment, each stage in the at least one stage includes a local self-attention module (e.g. component, code block, etc.) that extracts, per local window of a plurality of local windows within the input image, local features from the local window”; Para. 58 “As shown, local self-attention is computed on feature patches within the same local window only”]; determining an offset window that is offset (shifted) from the self-attention window (local window) [Para. 4 “computing self-attention within a local window of image patches limits the context in which an image patch is processed. In order to cross-interact with other regions (non-local windows) of the image, the windows must be shifted and the self-attention recomputed, which is computationally expensive”]; and
Hat also teaches extracting according to the offset window that is offset (shifted) [Para. 4 In order to cross-interact with other regions (non-local windows) of the image, the windows must be shifted and the self-attention recomputed, which is computationally expensive].
However, Hat doesn’t explicitly teach the rest of claim limitations.
YUAN teaches extracting an intermediate feature according to the offset window (shifted windows approach) to obtain a feature map (hierarchical feature maps) outputted at the layer [Para. 29 “For example, the shifted windows approach (e.g., overlapping, sliding windows of data) of the swin transformer may be used to look at the local attention, thereby saving memory when compared to traditional transformers that run full attention mechanisms”; Para. 30 “The swin transformer is one example of a vision transformer that builds hierarchical feature maps by merging image patches in deeper layers”; and 33 “Based on the hierarchical structure of the image encoder, feature pyramids can be output from the different scale levels”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat’s vision transformer local window self-attention by incorporating Yuan’s teaching of OFFSET window to recompute local attention over shifted windows and output feature map at processing levels. This modification improves hat by enabling cross-window feature interaction while preserving local window self-attention efficiently, thereby improving contextual feature extraction.
Hat also teaches determining an image classification feature (embedding) based on the combined-processed feature map (feature representations) using the image classification model (vision transformer) [Para. 53 “Resulting features output from the final stage 304D are passed through an average pooling layer 310 and then a linear layer 312 to create an embedding for a downstream task.”]; performing visually induced feeling-based classification on the sample (to-be-classified)/(input image) image according to the image classification feature (feature representation/embedding),[Para. 39 “In this case, the feature representations may be processed by the downstream task for performing image classification, object detection, instance segmentation, semantic segmentation, or any other desired computer vision-related task for the input image.”].
However, Hat in view of YUAN doesn’t explicitly teach the rest of claim limitations.
Chang teaches to obtain a classification result of the visually induced (visual sentiment analysis) feeling-based classification [Para. 9 “In one embodiment of the disclosed subject matter, techniques for visual sentiment analysis are provided”, “In an example embodiment, the disclosed subject matter provides a method for determining one or more viewer affects evoked from visual content using visual sentiment analysis.”, and “The method includes detecting one or more of the plurality of publisher affect concepts present in selected visual content, and determining, by the processor using the correlation model, one or more of the plurality of viewer affect concepts corresponding to the one or more of the detected publisher affect concepts”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat’s training framework, modified by YUAN, by using Chang’s visually induced feeling-based classification (visual sentiment analysis) output, namely viewer affect concepts, as the classification result used in Hat’s supervised weight-adjustment training process. This medication improves Hat by specializing the trained image model to viewer affect image classification, thereby producing a trained model for visually induced feeling-based classification rather than only generic image classification.
Hat also teaches updating a model parameter (weights) of the image classification model based on the classification result to obtain a target image classification model after training (trained neural network 1008) [Para. 84 “during training, data flows through the DNN in a forward propagation phase until a prediction is produced that indicates a label corresponding to the input. If the neural network does not correctly label the input, then errors between the correct label and the predicted label are analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset.”; Para. 95 “In at least one embodiment, training framework 1004 adjusts weights that control untrained neural network 1006”. Para. 95 “in at least one embodiment, training framework 1004 trains untrained neural network 1006 until untrained neural network 1006 achieves a desired accuracy.”; Para. 95 “In at least one embodiment, trained neural network 1008 can then be deployed to implement any number of machine learning operations”].
However, Hat in view of YUAN doesn’t explicitly teach updating model based on classification result of the visually induced feeling-based classification.
Chang teaches updating model based on classification result (viewer affect concepts) of the visually induced feeling-based classification (visual sentiment analysis) [Para. 9 “In one embodiment of the disclosed subject matter, techniques for visual sentiment analysis are provided”, “In an example embodiment, the disclosed subject matter provides a method for determining one or more viewer affects evoked from visual content using visual sentiment analysis.”, and “The method includes detecting one or more of the plurality of publisher affect concepts present in selected visual content, and determining, by the processor using the correlation model, one or more of the plurality of viewer affect concepts corresponding to the one or more of the detected publisher affect concepts”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat’s training framework, modified by YUAN, by using Chang’s visually induced feeling-based classification (visual sentiment analysis) output, namely viewer affect concepts, as the classification result used in Hat’s supervised weight-adjustment training process. This medication improves Hat by specializing the trained image model to viewer affect image classification, thereby producing a trained model for visually induced feeling-based classification rather than only generic image classification.
Regarding claim 9, Hat teaches wherein the window size of the self-attention window (local window) and the size of a feature patch (patches) in the target feature map inputted at the layer satisfies a size matching relationship [Para. 31 and 58]; and wherein extracting an intermediate feature (local features) of a covered feature patch (feature patches) according to the self-attention window [Para. 31 and 58] comprises:
Yuan teaches sequentially moving (shifted windows approach) the self-attention window (local window) in the target feature map, and respectively extracting a moving window feature of each feature patch (feature patches) covered by the self-attention window during the movement [Para. 29-30];
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat’s local self-attention processing by using Yaun’s shifted windows approach to shift the local attention window during local attention computation. This modification improves Hat by enabling cross-window interaction while preserving local-window attention efficiency, thereby improving contextual feature extraction.
Hat in view Yuan teaches doesn’t explicitly teach the rest of claim limitations.
LI teaches performing residual fusion (skip connections) on the moving window feature to obtain a fused moving window feature [Para. 27]; and sequentially performing fully-connected mapping and residual fusion on the fused moving window feature, to obtain the intermediate feature [Para. 29-31].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat in view of Yuan’s local self-attention processing by using Li’s skip connections to the output of local attention processing. This modification improves Hat by enabling cross-window interaction while preserving local-window attention efficiency, thereby improving contextual feature extraction.
Regarding claim 10, Hat doesn’t explicitly teach the claim limitation.
However, Yuan teaches wherein extracting the intermediate feature according to the offset window (shifted windows approach) to obtain a feature map outputted at the layer (hierarchical feature maps) comprises: sequentially moving the offset window (shifted windows approach) in the intermediate feature, and respectively extracting an offset window feature of a feature patch (image patches) covered by the offset window during the movement [Para. 29-30].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat’s local self-attention processing by using Yaun’s shifted windows approach to shift the local attention window during local attention computation. This modification improves Hat by enabling cross-window interaction while preserving local-window attention efficiency, thereby improving contextual feature extraction.
Hat in view Yuan teaches doesn’t explicitly teach the rest of claim limitations.
Li teaches performing residual fusion on the offset window feature to obtain a fused offset window feature [Para. 27]; and sequentially performing fully-connected mapping and residual fusion on the fused offset window feature, to obtain the feature map outputted at the layer [Para 29-31, fig. 2 and related description]. merging, at each layer starting from the second layer of the plurality of layers, when the layer meets a feature patch merging condition, feature patches in the feature map outputted at the previous layer, to obtain a merged feature map layer [Para 29-31, fig. 2 and related description]; and performing combined processing at the layer on the merged feature map, to obtain the feature map outputted at the layer [Para 29-31, fig. 2 and related description].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Hat in view of Yuan’s local self-attention processing by using Li’s skip connections to the output of local attention processing. This modification improves Hat by enabling cross-window interaction while preserving local-window attention efficiency, thereby improving contextual feature extraction.
Regarding claim 15, Hat in view of Yuan further in view of Chang teaches the claim limitations.
Furthermore, Chang teaches obtaining a visually induced feeling-based classification result (viewer affect concepts) of the to-be-classified image (visual content), and determining, according to the visually induced feeling-based classification result, a visually induced feeling attribute of content to which the to-be-classified image belongs [Para. 9, fig. 2 and related description]; determining attribute information (metadata) of the content (visual content), and updating the visually induced feeling attribute to the attribute information [Para. 11]; and determining account information of an account, and recommending content to the account based on the attribute information of the content and the account information [Para. 62, fig. 3, 5 and related description].
Regarding claim 18, Hat teaches a computer device, comprising a memory and a processor, the memory storing computer-readable instructions, and when executing the computer-readable instructions, the processor is configured to implement operations of the method according to claim 1 [Para. 87-89].
Regarding claim 19, Hat teaches a computer-readable storage medium, having computer-readable instructions stored therein, and when being executed by a processor, the computer-readable instructions are configured implement operations of the method according to claim 1 [Para. 87-89].
Regarding claim 20, Hat teaches a computer program product, having computer-readable instructions stored therein, and the instructions, when being executed by a processor, are configured to implement operations of the method according to claim 1 [Para. 87-89].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-0101 (IN USA OR CANADA) or 571-272-1000.
/SOLOMON G BEZUAYEHU/ Primary Examiner, Art Unit 2666
Prosecution Insights