DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The drawing objections have been withdrawn in light of the amended claims.
Applicant’s arguments with respect to claim(s) 1-6, 8, and 10-16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
The examiner would additionally like to address and clarify some of the Applicant’s remarks to potentially further prosecution.
The Applicant on Remarks page 7 states “The Examiner relies on Tandon for the extraction of feature vectors by a machine- learning model and turns to Salman only for the auxiliary teaching that a CNN can have "weights pre-trained on a large visual database (such as Imagenet)." "Then, for each label representation image, a convolutional neural network with weights pre-trained on a large visual database (such as Imagenet) is used to extract a feature vector," Salman, paragraph [0147]. However, Tandon expressly trains its classification models on the target biological samples themselves, as reflected in the very passages the Examiner cites for the classification step. See, e.g., "The reduced dimensional output describing the individual images and the labels associated with those images are provided to a random forests model generator 707 that produces a random forests model 709, which is ready to classify biological samples," Tandon, paragraph [0258]. The proposed combination therefore still involves training/retraining on target-particle images and does not disclose or suggest classifying target particles without such retraining”.
The Applicant amended claim 1 to explicitly recite “extraction of a feature map of said target particle from the input image by a convolutional neural network trained beforehand on a public image database, without retraining said convolutional neural network on images of the target particle”. Through the broadest reasonable interpretation, the Applicant is reciting a negative limitation in which the images of the target particle are not reinputted into the CNN. Although Tandon does recite using training input images, they do not teach retraining the model by using the same images of the target particles. While Tandon paragraph [0258] describes a process for training the model, the steps that occur in it are still applied when executing the model, as stated in paragraph [0338] “When the model is actually executed in the field, any data processed, which data is typically going to be include cellular artifacts having, for example, 50 pixel by 50 pixel regions, is subject to the same dimensionality reduction that was employed in the randomly selected cellular artifacts used to train the model”. Simply having an explanation as to how their model was trained does not inherently mean Tandon reinput the same images of the target particles into their model, as asserted by the Applicant.
“Salman does not cure this deficiency. Salman teaches pre-training only as a
starting point for extracting feature vectors; it does not teach abstaining from any further
training on the target-particle images. Neither Tandon nor Salman recognizes the
advantage - highlighted in the present specification - of avoiding retraining on the
target particle so that the same generic CNN can be reused across arbitrary particle types,
with class discrimination deferred entirely to the t-SNE embedding space”.
As stated above, Tandon does not teach retraining the model using the same images of the target particle. The modification is simply using a public database in place of Tandon’s dataset to accomplish the goal of feature extraction.
“The Examiner's stated motivations - "using a pre-trained model will skip training
and speed up the process of extraction" for Salman, and "t-SNE ensures data points are
close together, allowing for easier identification of clusters" for Song - do not bridge the
gap. Neither motivation would lead a person of ordinary skill to (a) forgo retraining on the
target particle images (Tandon's central teaching runs the other direction) or (b) replace
Tandon's PCA-plus-random-forests classifier with an in-t-SNE-space classifier operating
on a shared embedding of reference and target feature maps. Absent such teaching, the
rejection reflects impermissible hindsight reconstruction of the claim from Applicant's own
disclosure”.
In response to applicant's argument that the examiner's conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971).
On page 12 of Remarks, the Applicant states “Knight's analysis of optimal dimensionality reduction techniques also does not pertain with the current modification/motivation at hand, as while it states UMAP may be superior in construction of latent spaces, they do not say t-SNE is suboptimal. Knight's analysis also does not relate to the modification/motivation, as the modification is simply using the k-nearest neighbor algorithm to perform the classification of Tandon's invention. Extracting the "concept" of a known algorithm from an unrelated domain (scRNA-seq expression data in a UMAP space) and asserting that it can be dropped into a t-SNE embedding of image-derived particle feature maps is precisely the kind of unsupported generalization § 103 does not sanction. There must be an articulated, evidence-based reason why a person of ordinary skill would (i) build the common t-SNE embedding space required by claim 1, and then (ii) select k-nearest neighbor from the many alternatives listed in Knight to operate within that space”. Applicant’s arguments pertaining to Knight are moot due to the change in references in the rejection below. However, the examiner would like to clarify that using the k-nearest neighbors and t-SNE algorithm together would be a technique utilized by one of ordinary skill in the art. As t-SNE is a dimensionality reduction technique that provides a visualization of data points, the k-NN algorithm is a practical classification method for the t-SNE embedding, as it is a method that makes decisions based on nearby datapoints. This is additionally supported by Wikipedia articles pertaining to t-SNE and k-NN, as the first figure is an example of a t-SNE embedding and the second screenshot is an example of k-NN algorithms being used on a data set.
PNG
media_image1.png
389
348
media_image1.png
Greyscale
PNG
media_image2.png
626
943
media_image2.png
Greyscale
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6-8, 11-16 are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (US 20180211380 A1) in view of Salman (US 20210406644 A1) and Song (US 20190108444 A1).
Regarding claim 1, Tandon discloses a method for classifying at least one input image representing a target particle in a sample (“In the process 500, the one or more processors are configured to receive one or more images of a biological sample captured by the camera,” Tandon, paragraph [0200]).
While Tandon discloses extraction of a feature map of said target particle from the input image by a convolutional neural network without retraining said convolutional neural network on images of the target particle ("In some implementations, applying the machine-learning classification model to the plurality of images of cellular artifacts to classify the cellular artifacts includes: applying, by the one or more processors, a principal component analysis to the plurality of images of cellular artifacts to obtain a plurality of feature vectors for the plurality of cellular artifacts," Tandon, paragraph [0029]), they do not teach trained beforehand on a public image database.
However, Salman teaches a convolutional neural network trained beforehand on a public image database (“Then, for each label representation image, a convolutional neural network with weights pre-trained on a large visual database (such as Imagenet) is used to extract a feature vector,” Salman, paragraph [0147]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to pre-train Tandon’s model based on a public database, as taught by Salman.
The suggestion/motivation for doing so would have been because using a pre-trained model will speed up the process of extraction while the public database will provide a variety of training samples for a complete model.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
While Tandon in view of Salman discloses dimensionality reduction (“In the following description, two primary implementations of machine learning model will be presented: a convolutional neural network and a randomized Principal Component Analysis (PCA) random forests model,” Tandon, paragraph [0294]), they do not teach “reduction of a number of variables of the extracted feature map, by a t-SNE algorithm, wherein the t-SNE algorithm defines an embedding space common to a plurality of feature maps of reference particle and to the extracted feature map”.
However, Song teaches reduction of a number of variables of the extracted feature map, by a t-SNE algorithm, wherein the t-SNE algorithm defines an embedding space common to a plurality of feature maps of reference particle and to the extracted feature map ("In order to understand the behavior of the representations generated by different approaches, the t-SNE (T-distributed Stochastic Neighbor Embedding) algorithm can be used to obtain 2-D visualizations of the considered baselines and the described approaches. FIG. 5(a) illustrates a two-dimensional (2D) t-SNE visualization of the representation obtained for the non-plant dataset using the base kernel (Kernel 5)", Song, paragraph [0096]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to apply a t-SNE algorithm onto Tandon’s (in view of Salman) feature vectors, as taught by Song.
The suggestion/motivation for doing so would have been because t-SNE ensures data points are close together, allowing for easier identification of clusters.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Tandon in view of Salman and Song discloses unsupervised classification (“Training methods in which the identities of sample features and/or conditions are not used in training are termed unsupervised learning processes. While both supervised and unsupervised learning may be employed with the disclosed processes and systems, most examples herein are provided in the context of supervised learning,” Tandon, paragraph [0214]) of said input image depending on said feature map having a reduced number of variable, by implementation of an algorithm within said embedding space (“The reduced dimensional output describing the individual images and the labels associated with those images are provided to a random forests model generator 707 that produces a random forests model 709, which is ready to classify biological samples,” Tandon, paragraph [0258]).
Therefore, it would have been obvious to combine Tandon in view of Salman and Song to obtain the invention as specified in claim 1.
Regarding claim 6, Tandon in view of Salman and Song discloses the method as claimed in claim 1, wherein said feature map is a vector of numerical coefficients each associated with one elementary image of a set of elementary images each representing a reference particle ("In some implementations, randomized PCA generates a ten dimensional feature vector from each image in the training set. Every element in the training set is represented by this multi-dimensional vector, and fed into the random forests module to correlate between the label and the features," Tandon, paragraph [0342]), step of extracting said input image from an overall image of the sample comprises determination of numerical coefficients such that a linear combination of said elementary images weighted by said coefficients approximates the representation of said target particle in the input image (“ FIG. 28A shows hypothetical dataset having only two dimensions on the left and the hypothetical decision tree on the right that is trained from the hypothetical dataset. In this simplified illustrative example, each feature vector includes only two components; curvature and eccentricity. Each data point (or sample feature) is labeled as either 1 (feature of interest) or 0 (not feature of interest). Plotted on the x-axis on the left of the figure is curvature expressed in an arbitrary unit,” Tandon, Col. 19, paragraph [0346], Fig. 28A below, the location of the feature vector is a representation of the target particles).
PNG
media_image3.png
418
584
media_image3.png
Greyscale
Regarding claim 8, Tandon in view of Salman and Song discloses the method as claimed in claim 1, wherein the feature map has a reduced number of variables being the result of embedding the extracted feature map into said embedding space (Song, paragraph [0096], Fig. 5a below, "In order to understand the behavior of the representations generated by different approaches, the t-SNE (T-distributed Stochastic Neighbor Embedding) algorithm can be used to obtain 2-D visualizations of the considered baselines and the described approaches. FIG. 5(a) illustrates a two-dimensional (2D) t-SNE visualization of the representation obtained for the non-plant dataset using the base kernel (Kernel 5)", t-SNE is a dimensionality reduction technique).
PNG
media_image4.png
592
513
media_image4.png
Greyscale
Claim 11 corresponds to claim 1, additionally reciting “A system for classifying at least one input image representing a target particle in a sample” (“In various embodiments, the system includes at least one hardware component and/or at least one software component,” Tandon, paragraph [0382]),
Comprising at least one client comprising data-processor (“The system includes: a camera configured to capture one or more images of the biological sample; and one or more processors communicatively connected to the camera”, Tandon, paragraph [0004]). Thus, claim 11 is rejected for the same reasons of obviousness as claim 1.
Regarding claim 12, Tandon in view of Salman and Song discloses the system as claimed in claim 11, further comprising an observing device configured to determine said target particle in the sample. ("In some implementations, the one or more actuators can move the camera 412 and/or the stage 414 in one, two, or three dimensions,” Tandon, paragraph [0195]).
Regarding claim 13, Tandon in view of Salman and Song discloses a non-transitory computer storage medium comprising code instructions for executing a method as claimed in claim 1 (“An additional aspect of the disclosure relates to a non-transitory computer-readable medium storing computer-readable program code to be executed by one or more processors, the program code including instructions to cause a system including a camera and one or more processors communicatively connected to the camera to,” Tandon, paragraph [0038]),
for classifying at least one input image representing a target particle in a sample, when said program is executed on a computer. (“In the process 500, the one or more processors are configured to receive one or more images of a biological sample captured by the camera,” Tandon, paragraph [0200]).
Regarding claim 14, Tandon in view of Salman and Song discloses a non-transitory storage medium readable by a piece of computer equipment, on which a computer program product comprises code instructions for executing a method as claimed in claim 1 for classifying at least one input image representing a target particle in a sample. (“An additional aspect of the disclosure relates to a non-transitory computer-readable medium storing computer-readable program code to be executed by one or more processors, the program code including instructions to cause a system including a camera and one or more processors communicatively connected to the camera to,” Tandon, paragraph [0038]).
Regarding claim 15, Tandon in view of Salman and Song discloses the method as claimed in claim 1.
Tandon in view of Salman and Song does not teach “wherein the unsupervised classification comprises implementation of a k-nearest neighbor algorithm as the algorithm in the embedding space.”.
However, Song additionally teaches wherein the unsupervised classification comprises implementation of a k-nearest neighbor algorithm as the algorithm in the embedding space. ("To this end, the selection process can be repeated with different clusterings techniques, such as the k-means, k-medians, k-medoids, agglomerative clustering, and/or spectral clustering based on k nearest neighbors", Song, paragraph [0073]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to implement the k-nearest neighbor algorithm in Tandon’s (in view of Salman and Song) embedding space, as additionally taught by Song.
The suggestion/motivation for doing so would have been because the algorithm requires no training, thus reducing processing and saving time.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Tandon in view of Salman and the additional teachings of Song to obtain the invention as specified in claim 15.
Claim 16 corresponds to claim 15, additionally reciting the system (Tandon, paragraph [0004], “One aspect of the disclosure relates to a system for identifying a sample feature of interest in a biological sample of a host organism”). Thus, it is rejected for the same reasons of obviousness as claim 15.
Claim(s) 2-3, 5 are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (US 20180211380 A1) in view of Salman (US 20210406644 A1), Song (US 20190108444 A1) and in further view of Douet (US 10831156 B2).
Regarding claim 2, Tandon in view of Salman and Song discloses the method as claimed in claim 1.
Tandon in view of Salman and Song does not teach “wherein the particles are represented in a uniform manner in the input image and in each elementary image, and in particular centered on and aligned in a predetermined direction.”.
However, Douet teaches wherein the particles are represented in a uniform manner in the input image and in each elementary image, and in particular centered on and aligned in a predetermined direction ("Without being bound by theory, the inventors were able to observe the occurrence of two poles in a bacterium before its division, which might correspond to the meiosis of a bacterium, as illustrated in FIG. 8 which shows the time variation of a bacterium," Douet, Col. 15, Line 9-14, Fig. 8 below).
PNG
media_image5.png
323
511
media_image5.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to capture thumbnail images of Tandon’s (in view of Salman and Song) samples in a uniform and centered manner, as taught by Douet.
The suggestion/motivation for doing so would have been to acquire data of samples with less position variance, resulting in faster machine learning and less room for classification errors.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Tandon in view of Salman, Song and in further view of Douet to obtain the invention as specified in claim 2.
Regarding claim 3, Tandon in view of Salman, Song, and Douet discloses the method as claimed in claim 2, comprising a step of extracting said input image from an overall image of the sample, so as to represent said target particle in said uniform manner. ("Typically, the disclosed embodiments have enabled to obtain a thumbnail image representing a bacterium with a number of pixels in the range from 100 to 400 pixels," Douet, Col. 15, Line 3-5, Fig. 8 above, the sample images taken above are from an image of samples).
Regarding claim 5, Tandon in view of Salman, Song, and Douet discloses the method as claimed in claim 3, wherein step of extracting said input image from an overall image of the sample comprises obtaining said overall image from an intensity image of the sample, said image being acquired ("The system automatically moves the camera 428 and/or the stage 430 so that the camera 428 can capture one or more images of the biological sample without requiring a human operator to adjust the camera or the stage to change the relative positions of the camera 428 and the biological samples on the stage 430", Tandon, paragraph [0196]).
Claim(s) 4 is rejected under 35 U.S.C. 103 as being unpatentable over Tandon (US 20180211380 A1) in view of Salman (US 20210406644 A1), Song (US 20190108444 A1), Douet (US 10831156 B2), and in further view of Abu Qura (US 20210334971 A1).
Regarding claim 4, Tandon in view of Salman, Song, and Douet discloses the method as claimed in claim 3, wherein step of extracting said input image from an overall image of the sample comprises segmentation of said overall image so as to detect said target particle in the sample (“The one or more processors are configured to: receive the one or more images of the biological sample captured by the camera; segment the one or more images of the biological sample to obtain a plurality of cellular artifacts,” Tandon, paragraph [0004]).
Tandon in view of Salman, Song, and Douet does not teach “then cropping of the input image to said detected target particle”.
However, Abu Qura teaches cropping of the input image to said detected target particle (“The processor may perform various operations on the image to make the image cleaner for analysis. For example, the processor may automatically crop the image 240 or perform edge detection of the image 240 of the test kit 210 prior to transmitting the image for analysis,” Abu Qura, paragraph [0068]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to crop the segmented image of Tandon (in view of Salman, Song, and Douet), as taught by Abu Qura.
The suggestion/motivation for doing so would have been to acquire a smaller image, resulting in lesser data processing and resources.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Tandon in view of Salman, Song, Douet, and in further view of Abu Qura to obtain the invention as specified in claim 4.
Claim(s) 10 are rejected under 35 U.S.C. 103 as being unpatentable over Tandon (US 20180211380 A1) in view of Salman (US 20210406644 A1), Song (US 20190108444 A1), and in further view of Lee (US 20210256322 A1).
Regarding claim 10, Tandon in view of Salman and Song discloses the method as claimed in claim 1, for classifying a sequence of input images representing said target particle in a sample over time ("In the process 500, the one or more processors are configured to receive one or more images of a biological sample captured by the camera,” Tandon, paragraph [0200], taking more than one image of a sample is taking images over time).
Tandon in view of Salman and Song does not teach “wherein step of extraction of a feature map comprises concatenation of the extracted feature maps of each input image of said sequence”.
However, Lee teaches wherein step of extraction of a feature map comprises concatenation of the extracted feature maps of each input image of said sequence ("The first to fourth fully-connected layers 831, 832, 833, and 834 may perform parallel processing (e.g., simultaneous processing) on the first to fourth partial images. The feature vectors of the first to fourth partial images may be concatenated to form one vector (hereinafter, a connected feature vector),” Lee, paragraph [0098]).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use Lee’s method of concatenation on Tandon’s (in view of Salman and Song) feature vectors to form a connected vector.
The suggestion/motivation for doing so would have been to merge the information of different feature vectors, allowing for more comprehensive results.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Tandon in view of Salman and Song and in further view of Lee to obtain the invention as specified in claim 10.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WAYNE ZHANG whose telephone number is (571) 272-0245. The examiner can normally be reached Monday-Friday 10:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WAYNE ZHANG/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights