DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8, 10-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sia (WO-2022076516-A1) in view of Axel (Axel Klawonn et al. Model Parallel Training and Transfer Learning for Convolutional Neural Networks by Domain Decomposition. Aug. 2024, https://doi.org/10.48550/arXiv.2408.14442) and Hu (US-11908580-B2).
Regarding Claim 1, representative of Claims 10 and 19, Sia teaches a method, implemented by a computing system, of training a computer vision model to classify health test kit results, the method comprising:
obtaining a plurality of training images, wherein each training image depicts a plurality of health test results in respective segments of a test membrane of a health test kit ([0063]: learning the optimal parameters of the feature-extraction network and the classifier module by minimizing the loss functions given a set of training images for an assay kit, [0062]: image of an LFA kit, which enters a cloud-hosted pipeline with an instance-segmentation model that corrects the orientation and perspective of the raw image, segments the assay kit from background and the membrane from assay kit, and crops individual zones);
obtaining, for each training image, labeling indicating the health test results depicted by the training image ([0063]: the neural network model pre-trained on a set of labeled LFA images);
([0089] We used the ResNetl8 CNN as the feature extractor, [0062]: images of zones enter a feature-extraction network, which is learned in order to generate robust feature representation as unique signatures to discriminate positive cases from negative cases, [0062]: From latent feature vectors for each zone, a binary classifier recognizes colored rectangular bands, the form factor seen in the vast majority of LFAs (References 32, 33), and determines whether a band is present or absent in each zone) based on:
local features extracted by the local CNN from the respective one of the segments of the training image ([0062]: From latent feature vectors for each zone, a binary classifier recognizes colored rectangular bands, the form factor seen in the vast majority of LFAs (References 32, 33), and determines whether a band is present or absent in each zone. See Fig. 2D. Examiner notes it appears that the feature extractor is applied to each zone);
Sia does not explicitly teach training a plurality of local Convolutional Neural Networks (CNNs) of the computer vision model in parallel.
Axel teaches a plurality of local Convolutional Neural Networks (CNNs) of the computer vision model in parallel ([abstract]: for each of these sub images, local CNNs with a proportionally smaller number of parameters are trained in parallel).
Although Sia teaches a training image of a test membrane in at least [0052] and Fig. 2D, neither Sia nor Axel teaches global features extracted by a global CNN of the computer vision model.
Hu teaches global features extracted by a global CNN of the computer vision model ([abstract]: device extracts a global feature from the medical images using a second network model. The device fuses the extracted local feature and the extracted global feature to form a fused feature. The device performs image classification based on the fused feature).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the teachings of Sia to include the teachings of Axel by substituting the use of a single feature extractor CNN repeatedly used per zone for multiple CNNs in parallel per zone/sub image. Doing so would improve processing speed for classification. Further it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the Sia and Axel combination to include the teachings of Hu, by including global feature extraction and incorporation of both global and local features in a classification. Doing so would improve the accuracy of classification (Hu, [0062]).
Regarding Claim 2, representative of Claim 11, the Sia, Axel and Hu combination, hereinafter referred to as the Sia combination, teaches the method of claim 1. In addition, Sia teaches wherein training the
for each segment of the test membrane of the training image, using the local CNN corresponding to the segment to: generate a local feature map indicating the local features extracted from the segment ([0095]: individual zones are cropped, and passed through a software pipeline consisting of a feature extractor followed by a binary classifier, [0089] We used the ResNetl8 CNN as the feature extractor);
[0095]: individual zones are cropped, and passed through a software pipeline consisting of a feature extractor followed by a binary classifier. Classification of each zone allows, via a kit-specific lookup table, for a final classification of assay result (“kit-level” classification or result) as positive, negative, or invalid.)
Axel teaches training the plurality of local CNNs ([abstract]: for each of these sub images, local CNNs with a proportionally smaller number of parameters are trained in parallel. Examiner notes see the combination rationale of Claim 1),
Hu teaches generating, by the global CNN, a global feature map indicating the global features extracted from the training image ([0005]: extracting a local feature (e.g., a local medical feature) from the texture image using a first network model; extracting a global feature (e.g., a global medical feature) from the medical images using a second network model, [0052]: second network model may also belong to the VGG network, the Inception network, the ResNet network, the DenseNet network, or the like); and
generate a combined feature map by combining the local feature map with the global feature map ([0005]: fusing the extracted local feature and the extracted global feature to form a fused feature);
extract combined features from the combined feature map ([0122]: In an embodiment, S808 may further include: The terminal fuses the global training medical feature and the local training medical feature by using the fully connected layer, to obtain the fused feature; performs convolution calculation on the fused feature); and
generate a revised local feature map indicating the combined features extracted from the combined feature map, wherein predicting the health test result depicted in the segment based on the local features and the global features comprises using the revised local feature map to generate a prediction based on the combined features ([0122]: In an embodiment, S808 may further include: The terminal fuses the global training medical feature and the local training medical feature by using the fully connected layer, to obtain the fused feature; performs convolution calculation on the fused feature; and performs image classification on the medical image samples based on a result obtained after the convolution calculation).
Regarding Claim 3, representative of Claim 12, the Sia combination teaches the method of claim 2. Sia teaches wherein training the ([0007] In some embodiments, the feature extractor also has been pre-trained by: using a classifier to classify the feature-extracted images; determining a second calculated loss between the classified images and labels for the images in the inputted set of images; and using the second calculated loss to update parameters of the feature extractor).
Axel teaches the plurality of local CNNs ([abstract]: for each of these sub images, local CNNs with a proportionally smaller number of parameters are trained in parallel, [section 3.1, paragraph 1]: All networks are trained using the Adam (Adaptive moments) optimizer [4] and the cross-entropy loss function).
Regarding Claim 4, representative of Claim 13, the Sia combination teaches the method of claim 3. In addition, Sia teaches wherein:
using the local CNN of each segment to generate the prediction of the health test result depicted in the segment comprises using a binary classifier of the local CNN to generate the prediction ([0062]: From latent feature vectors for each zone, a binary classifier recognizes colored rectangular bands, the form factor seen in the vast majority of LFAs (References 32, 33), and determines whether a band is present or absent in each zone; and
training the ([0090] Given a training set, all the images were fed into the model in sequence and the loss was calculated for both gradient backpropagation and for updating the model).
Axel teaches the plurality of local CNNs ([abstract]: for each of these sub images, local CNNs with a proportionally smaller number of parameters are trained in parallel, [section 3.1, paragraph 1]: All networks are trained using the Adam (Adaptive moments) optimizer [4] and the cross-entropy loss function).
Regarding Claim 5, representative of Claim 14, the Sia combination teaches the method of claim 4. In addition, Sia teaches wherein:
the loss function comprises a cross-entropy loss function ([0065]: the classifier was trained to learn specific prototypes associated with the positive and negative classes using cross-entropy (CE) loss);
backpropagating the loss function comprises adjusting weights applied by each of the binary classifiers to generate the predictions ([0123]: the annotated classification label and the self-generated edge detection label, both the cross-entropy (CE) loss and the mean-squared-error (MSE) may be used as the objective, and gradient backpropagation may be performed for parameter updating).
Regarding Claim 6, representative of Claim 15, the Sia combination teaches the method of claim 1. In addition, Sia teaches further comprising:
obtaining a non-training image depicting a plurality of actual health test results in the respective segments of the test membrane of the health test kit ([0074]: a random set of images of each class were selected from the training dataset for model adaptation, and performance of the trained model was validated against a separate evaluation dataset. Examiner interpreting validation dataset as non-training images); and
using [0095]: individual zones are cropped, and passed through a software pipeline consisting of a feature extractor followed by a binary classifier. Classification of each zone allows, via a kit-specific lookup table, for a final classification of assay result (“kit-level” classification or result) as positive, negative, or invalid).
Axel teaches using trained local CNNs ([abstract]: for each of these sub images, local CNNs with a proportionally smaller number of parameters are trained in parallel).
Hu teaches incorporating global CNN ([abstract]: device extracts a global feature from the medical images using a second network model. The device fuses the extracted local feature and the extracted global feature to form a fused feature. The device performs image classification based on the fused feature).
Regarding Claim 7, representative of Claim 16, the Sia combination teaches the method of claim 6. In addition, Sia teaches further comprising further training at least one of the local CNNs responsive to receiving result labels indicating each of the actual health test results depicted by the non-training image ([0065]: the classifier was trained to learn specific prototypes associated with the positive and negative classes using cross-entropy (CE) loss, [0074]: a random set of images of each class were selected from the training dataset for model adaptation, and performance of the trained model was validated against a separate evaluation dataset. Examiner interpreting validation dataset as non-training images).
Regarding Claim 8, representative of Claim 17, the Sia combination teaches the method of claim 1. In addition, Sia teaches further comprising detecting the segments of each training image, wherein the segments of each training image are arranged in a single row ([0062]: (Fig. 2D), a user takes an image of an LFA kit, … segments the assay kit from background and the membrane from assay kit, and crops individual zones (i.e., regions in the membrane corresponding to bands and a portion of surrounding area). See Fig. 2D LFA kit has membrane containing and displaying bands in a single row).
Claim(s) 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sia (WO-2022076516-A1) in view of Axel (Axel Klawonn et al. Model Parallel Training and Transfer Learning for Convolutional Neural Networks by Domain Decomposition. Aug. 2024, https://doi.org/10.48550/arXiv.2408.14442), Hu (US-11908580-B2), and Anfossi (Anfossi L, Di Nardo F, Cavalera S, Giovannoli C, Baggiani C. Multiplex Lateral Flow Immunoassay: An Overview of Strategies towards High-throughput Point-of-Need Testing. Biosensors (Basel). 2018 Dec 26;9(1):2. doi: 10.3390/bios9010002.).
Regarding Claim 9, representative of Claim 18, the Sia combination teaches the method of claim 1. Although Sia teaches using lateral flow assays of different housing dimensions in [0061], none of the Sia combination explicitly teaches the remaining limitations of Claim 9.
Anfossi teaches further comprising detecting the segments of each training image, wherein the segments of each training image are arranged in a two-dimensional grid ([Section 1, paragraph 7]: the spatial separation of detection sites in one strip, the separation of reaction sites using individual strips arranged in an ‘array-like’ format. See Fig. 2b, multiple strips forming a 2D array).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have modified the Sia combination by substituting the single row lateral flow device with a multiplex lateral flow device as taught by Anfossi. Doing so would provide the predictable result of an image of a health test kit containing segments to use for classification.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANICE VAZ whose telephone number is (703)756-4685. The examiner can normally be reached Monday-Friday 9:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANICE E. VAZ/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights