DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
2. Claimed benefit of domestic priority to U.S. Provisional Patent Application No. 63/136,382, filed 12 January 2021 is acknowledged. In this action, all claims are examined as though they had an effective filing date 12 January 2021. In future actions, the effective filing date of one or more claims may change, due to amendments to the claims, or further analysis of the disclosure(s) of the priority application(s).
Information Disclosure Statement
3. The information disclosure statements (IDSs) submitted on 12 July 2023 and 29 November 2024 are being considered by the examiner. The two references lined out on the IDS submitted and 29 November 2024 are not being considered because they are duplicate entries of references on the previously submitted IDS document.
Drawings
4. The drawing submitted on 12 July 2023 are accepted by the examiner.
Claim Objections
5. Claim 5 is objected to because of the following informalities: Claim 5 recites: “the particular of embryo feature”, which is grammatically incorrect. A possible correction is to change the phrase to: “the particular embryo feature”.
Appropriate correction is required.
Claim Interpretation
6. Regarding the term ‘embryo’, according to the specification para. 0079-0081, “the specimen imaged includes oocytes or embryos and in particular embryos” and “the timing of optical inspection of the embryo may be performed within 10 days after fertilization”. The term ‘embryo’ in claims 1-17 and claims 23-24 is therefore interpreted to include all early stages after fertilization of an oocyte within 10 days after fertilization, including the first 4 days after fertilization, which includes the ‘zygote’ stage.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
7. Claims 18-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 2A, Prong 1
In accordance with MPEP § 2106, claims found to recite statutory subject matter (Step 1: YES) are then analyzed to determine if the claims recite any concepts that equate to an abstract idea, law of nature or natural phenomenon (Step 2A, Prong 1). In the instant application, the claims recite the following limitations that equate to an abstract idea:
Claim 18 recites: determine birefringence properties of the specimen based at least in part on the image data
Claim 18 recites: classify features of the polarized light image using a classifier
Claim 18 recites: identify features of the polarized light image as mitotic spindles
Claim 18 recites: determine mitotic activity of the specimen based at least in part on the identified mitotic spindles
Claim 18 recites: predict a ploidy status of the specimen based on the mitotic activity
Claim 19 recites: determine whether the mitotic activity is below a predetermined threshold, and when the mitotic activity is below the predetermined threshold the ploidy status of the specimen is predicted to be euploid
Claim 20 recites: the system of claim 18, wherein the mitotic activity of the specimen is determined based at least in part on a number of the identified mitotic spindles
Claim 21 recites: determine geometric shapes of the identified mitotic spindles
Claim 21 recites: the mitotic activity of the specimen is determined based at least in part on the geometric shapes of the identified mitotic spindles
Claim 22 recites: identify features of the polarized light image as an inner cell mass (ICM) and a trophectoderm (TE)
Claim 22 recites: determine locations of the identified mitotic spindles as in the ICM or in the TE
Claim 22 recites: the mitotic activity of the specimen is determined based at least in part on the locations of the identified mitotic spindles
The limitations regarding determining birefringence properties, classify features, determine mitotic activity, predict a ploidy status, identify features of the polarized light image, determine locations of the identified mitotic spindles are verbal equivalents that describe a mathematical calculation that is performed as the limitation (such as measuring the ratio between two distances in image data) and are so simple that they could be performed in the human mind or with pen and paper. Therefore, these limitations fall under the "Mathematical concepts" and "Mental processes" groupings of abstract ideas.
The limitations directed to ‘determine whether the mitotic activity is below a predetermined threshold’ and ‘determine geometric shapes’ are generically recited data analysis steps that can be practically performed in the human mind because the human mind is capable of identifying relevant information, comparing values, and determining information from other values. Therefore, these limitations fall under the "Mental processes" groupings of abstract ideas.
The limitations in claims 20-22 further limit the ploidy status that is predicted or the mitotic activity that is determined, simply further limit the abstract ideas but don’t change their position as abstract ideas.
While claim 18 recite performing some aspects of the analysis with a computer, there are no additional limitations that indicate that this processor requires anything other than carrying out the recited mental process or mathematical concept in a generic computer environment. Merely reciting that a mental process is being performed in a generic computer environment does not preclude the steps from being performed practically in the human mind or with pen and paper as claimed. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then if falls within the "Mental processes" grouping of abstract ideas. As such, claims 18-24 recite an abstract idea (Step 2A, Prong 1: YES).
Step 2A, Prong 2
Claims found to recite a judicial exception under Step 2A, Prong 1 are then further analyzed to determine if the claims as a whole integrate the recited judicial exception into a practical application or not (Step 2A, Prong 2). This judicial exception is not integrated into a practical application because the claims do not recite an additional element that reflects an improvement to technology or applies or uses the recited judicial exception in some other meaningful way. Rather, the instant claims recite additional elements that amount to mere instructions to implement the abstract idea in a generic computing environment or insignificant extra-solution activity. Specifically, the claims recite the following additional elements:
Claim 18 recites: a computer-implemented system comprising: an image sensor
Claim 18 recites: a processor in communication with the image sensor
Claim 18 recites: and a memory in communication with the processor
Claim 18 recite: the memory storing instructions that, when executed by the processor cause the processor to [perform the method]
Claim 18 recites: receive, from the image sensor, image data representing emerging polarized light that has traversed a specimen
Claim 18 recites: generate a polarized light image representative of the specimen based at least in part on the birefringence properties
Claim 19 recites: the system of claim 18, wherein the memory stores further instructions that, when executed by the processor cause the processor to [perform the method]
Claim 21 recites: the system of claim 18, wherein the memory stores further instructions that, when executed by the processor cause the processor to [perform the method]
Claim 22 recites: the system of claim 18, wherein the memory stores further instructions that, when executed by the processor cause the processor to [perform the method]
Claim 23 recites: the system of claims 18, wherein the specimen is from a mammal embryo
Claim 24 recites: the system of claim 23, wherein the mammal is a human
The limitations to ‘generate a polarized light image’ and ‘receive image data’ merely serve to gather data that is used an input for the judicial exception. Therefore, these limitations are mere data gathering activities. As set forth in MPEP 2106.05(g), mere data gathering activity has been identified by the courts as insignificant extra-solution activity that does not provide a practical application. The limitations of claims 23 and 24 that further limit the data that is gathered, but do not integrate the judicial exception into a practical application because they just further limit data gathering activities but don’t change their position as data gathering activities.
There are no limitations that indicate that the processor and memory require anything other than a generic computing system used in an ordinary capacity (e.g. to receive, store or transmit data). As such, these limitations equate to mere instructions to implement the abstract idea on a generic computer that the courts have stated does not render an abstract idea eligible in Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984.
There is no indication that the claimed image sensor is anything more than a generic camera configured to obtain microscopic images. Thus, this sensor does not render the abstract idea eligible as no indication is provided that the camera has any particular functionality (see MPEP 2106(b)).
The limitations directed to ‘storing instructions’ and ‘storing further instructions’ are insignificant extra-solution activities, which are incidental to the method and system and are merely nominal or tangential additions to the claims.
The above recited additional elements do not provide a practical application of the recited judicial exception. As such, claims 18-24 are directed to an abstract idea (Step 2A, Prong 2: NO).
Step 2B
Claims found to be directed to a judicial exception are then further evaluated to determine if the claims recite an inventive concept that provides significantly more than the judicial exception itself (Step 2B).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite additional elements that equate to mere instructions to apply the recited exception in a generic computing environment or well-understood, and conventional activity.
The limitations directed to data gathering (generate a polarized light image and receive image data) and those directed to data storage (storing instructions and storing further instructions) do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As set forth in MPEP section 2106.05(g), the courts have decided that limitations that merely add an insignificant extra-solution activity, do not amount to an inventive concept, particularly when the activities are well-understood and conventional. As set forth in MPEP section 2106.05(d), the courts have recognized that limitations directed to data gathering that are claimed as insignificant extra-solution activity are routine, well understood and conventional (Mayo Collaborative servs. V. Prometheus Labs., Inc., 566 U.S. at 79, 101 USPQ2d at 1968). Additionally, storing and retrieving information from memory has also been deemed well-understood, routine and conventional activity (Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
In addition, the specification indicates that the image sensor can be any suitable image capture component, for example a digital camera (with still and/or video recording capabilities) and can include CMOS or CCD sensors (para. 0078). Additionally, polarizing microscopy is well understood, routine and conventional as evidenced by Koike-Tani et al. (2015, Molecular Reproduction and Development, Vol. 82, p. 548-562). Koike-Tani et al. discloses that LC-PolScope, a birefringence imaging system with image acquisition and processing functions for use in imaging in reproductive and developmental biology was commercially available from Cambridge Research and Instrumentation, Inc., Hopkinton MA, now part of Perkin-Elmer, Waltham MA before the effective filing date (Fig. 5, p. 553, col. 1, para. 2 – col. 2, para. 2).
The limitations of claim 18, pertaining to the computer system used to execute the method, are directed to performing judicial exceptions with a generic computing system on a generic computer. These limitations are not sufficient to amount to significantly more than the judicial exception because, as set forth in the MPEP section 2106.05(d)(II)), using a generic computing environment or generic computer to perform the judicial exception, has been deemed well-understood, routine and conventional activity including receiving or transmitting data over a network (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362), performing repetitive calculations (Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012)), and storing and retrieving information in memory (Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)). Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception are insufficient to provide significantly more.
The additional elements do not comprise an inventive concept when considered individually or as an ordered combination that transforms the claimed judicial exception into a patent-eligible application of the judicial exception. Therefore, the claims do not amount to significantly more than the judicial exception itself (Step 2B: No). As such, claims 18-24 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
8. Claims 1-7, 10 and 13-17 are rejected under 35 U.S.C. 103 as being unpatentable over Hall et al. (WO2020198779A1; 7/12/2023 IDS document), in view of Tilia et al. (Fertility and Sterility, 2016, vol 105, p. 1085-2000) and Gu et al. (reproductive BioMedicine Online, Vol. 19, 2009, p. 745-754), as evidenced by Golubovsky (Human Reproduction, 2003, Vol. 18, p. 236-242). The italicized text corresponds to the instant claim limitations.
With respect to claims 1 and 17, Hall et al. teaches a method and system for computationally generating an AI model configured to estimate an embryo viability score from an image. Hall et al. teaches that the system comprises a processor and a memory comprising instructions to configure the processor to receive an image after in vitro fertilization, upload the image to an AI model configured to generate an embryo viability score from the image (para. 0007, 0032; A computer-implemented system/method, the system comprising: a processor, a memory in communication with the processor; the memory storing instructions that, when executed by the processor, cause the processor to [perform the method]).
Pertaining to claims 1 and 17, Hall et al. teaches that either a single image or multiple images are obtained of human embryos being considered for implantation and that the image(s) can be captured using polarized light microscopy (para. 0071-0072, 0082; receive polarized light image data reflective of a mammal embryo specimen).
Regarding claims 1 and 17, Hall et al. teaches that a convolutional neural network (CNN) deep learning model can process input images to generate an outcome probability, which can be a probability of embryo viability. Regarding the teaching of ‘probability’ versus ‘likelihood’, according to the specification of the instant application, a CNN classification metric output is reflective of a likelihood of the ploidy status (instant application para. 0008) (Hall et al., para. 0024; 0049; Fig. 8; present the polarized light image data to a convolutional neural network (CNN) trained to classify specimens; generate with the CNN a classification metric reflective of a likelihood).
With respect to claims 1, 2, and 16-17, Hall et al. is silent to: a system for classifying ploidy status, to classify [embryo] specimens according to a ploidy status and reflective of a likelihood of the ploidy status (claims 1 and 17) and the system of claim 1, wherein the ploidy status includes at least one of aneuploidy, mosaicism, or euploidy (claim 2); the system of claim 1, wherein the instructions, when executed by the processor cause the processor to generate the polarized light image data upon determining birefringence properties of the mammal embryo specimen (claim 16). However, these limitations were known in the art at the time of the effective filing date of the invention as taught by Tilia et al. and Gu et al.
Pertaining to claims 1 and 17, Tilia et al. teaches that a ploidy classifier can be made from collected polarized light microscopy images of spindle morphology. Tilia et al. teaches a method of determining embryo ploidy from a polarized light image and using this information for embryo selection in addition to conventional embryo morphologic parameters. Tilia et al. further discloses that oocyte spindle morphology analyzed from polarized light images is significantly associated with the resulting embryo’s ploidy (e.g. those with no visible spindle had significantly less probability of producing a euploid embryo). Tilia et al. further teaches generating a ROC curve for the prediction of ploidy status using spindle morphology, which inherently provides a point on the ROC curve that corresponds to the likelihood ratio for a single test value represented by that point (Tilia et al., p. 1086, col. 1, para. 2-4; p. 1089, col. 2, para. 2; Figure 1; abstract; supplementary Figure 1; for classifying ploidy status, to classify specimens according to a ploidy status and reflective of a likelihood of the ploidy status). However, Tilia et al. performs this classification using polarized light image data reflective of a mammal oocyte specimen not an embryo specimen. Gu et al. uses polarized light image data reflective of a mammal embryo specimen below.
Pertaining to claims 1 and 17, Gu et al. teaches observing the mitotic spindle morphology in human tripronuclear (3PN) zygotes using polarized light images. Gu et al. discloses that 3PN zygotes with bipolar mitotic spindles were observed (using Polscope) to cleave to 2-cell embryos and those with tripolar spindles to cleave to 3-cell embryos, demonstrating that abnormal mitotic spindles resulted in abnormal cleavage patterns. Gu et al. further discloses observing various abnormal spindles in human 3PN zygotes in the first cleavage such as tripolar spindles, multi-spindle and irregular-shape spindles.
To make predictions about the relationship of spindle morphology in 3PN zygotes to abnormal ploidy status of the resulting embryo, Gu et al. uses ploidy data from several prior studies summarized in Gu et al. and Golubovsky et al. and described below: It is known that the 3 variants of human 3PN zygote cleavage correspond to 3 outcomes: (i) mitotic division with biopolar spindle gives 3PN blastomeres and embryos (triploid) in 25% of cases, (ii) exclusion of one haploid genome from the metaphase plate of the first cleavage division (14–32% of cases), results in 2PN diploid, 2PN/3PN mosaics (diploid and triploid) and 1PN/2PN diploid derivatives; and (iii) in 50–60% of 3PN zygotes, a tripolar spindle is formed at the first cleavage division, resulting in dramatic abnormalities in chromosome distribution, resulting in aneuploidy.
Therefore, based on these disclosed data, zygotes with tripolar spindles that Gu et al. observed to cleave to 3-cell embryos, inherently correspond to aneuploidy in the embryo and zygotes with dipolar spindles that Gu et al. observed to cleave to 2-cell embryos, inherently correspond to heteroploidy or tripoloidy in the embryo. Gu et al. further predicts the ploidy status of abnormal mitotic spindles: In their experiment wherein the extra pronucleus is removed in 3PN zygotes, they predict that: 1) it is the bipolar mitotic spindle during the first cleavage that produces diploid embryos and 2) the percentage of 2n cells in the 2n/3n mixploid embryos (in outcome (ii) above) is increased, and diploidization increases potential to reach blastocyst stage (Gu et al. p. 750, col. 2, para. 2 – p. 753, col. 1, para. 1; Golubovsky Figure 1; classify embryo specimens according to a ploidy status).
Pertaining to claim 2, Gu et al. discloses that in the first cleavage, the human 3PN zygotes with bipolar mitotic spindles were seen to cleave to 2-cell embryos and human 3PN zygotes with tripolar mitotic spindles cleaved to 3-cell embryos. Based on prior studies discussed by Gu et al. and as evidenced by Golubovsky et al.: 1) tripolar spindle formation results in a mosaic embryo with three equally sized blastomeres at the first mitotic division, due to the random segregation of the sister chromatids of the three haploid sets of chromosomes to the three poles, which has karyotype instability leading to aneuploidy, and 2) dipolar spindle formation results in either triploidy or heteroploidy (including molecular mosaics) in the embryo (Gu et al., p. 746, col. 2, para. 1; p. 749, col. 2, para. 2; p. 750, col. 2, para. 2 – p. 753, col. 1, para. 2; Golubovsky et al., Fig. 1; the system of claim 1, wherein the ploidy status includes at least one of aneuploidy, mosaicism, or euploidy).
Regarding claim 16, Gu et al. discloses that the polarized light microscope uses novel electro-optical hardware and digital processing to image macromolecular structures in cells on the basis of their birefringence (an inherent physical property of highly ordered molecules such as microtubules). Gu et al. further discloses using these properties to non-invasively investigate spindle dynamics (p. 746, col. 2, para. 3; the system of claim 1, wherein the instructions, when executed by the processor cause the processor to generate the polarized light image data upon determining birefringence properties of the mammal embryo specimen).
An invention would have been prima facie obvious to one of ordinary skill in the art at the effective filing date of the invention if some motivation in the prior art would have led that person to combine the prior art teachings to arrive at the claimed invention. Gu et al. taught that mitotic spindle
abnormalities may be a major pathway leading to post zygotic chromosomal abnormalities, which may account for the developmental loss in human embryos. Gu et al. further teach that the polarized light microscopy provides a rapid, simplified and non-invasive method for identifying mitotic spindles in human embryos (p. 753, col. 2, para. 1). Therefore, one of ordinary skill in the art would have been motivated to utilize the method to detect mitotic spindle abnormalities of embryos from polarized light images and the relationship to ploidy status taught by Gu et al. and the method to predict a likelihood of ploidy status in an embryo from spindle morphology taught by Tilia et al. in the method and system for selecting viable embryos taught by Hall et al., in order to develop a noninvasive marker of embryo ploidy to help select embryos for transfer to prevent developmental loss. Furthermore, one of ordinary skill in the art would predict that the predictive association of ploidy status with spindle morphology taught by Gu et al. and Tilia et al. could be readily added to the system of Hall et al. with a reasonable expectation of success because they both pertain to selecting embryos for implantation using images acquired using polarized light microscopy. The invention is therefore prima facie obvious.
Regarding claim 3, Hall et al. discloses that a semantic segmentation model may be trained which identifies a class for each pixel in an image. Hall et al. further disclose object detection to identify a bounding box that contains all pixels associated with an object (which includes an object classifier head and a bounding box regression head) (para. 0082, 0085; 0091; Claim 3 recites: the system of claim 1, wherein the classification metric is received from a classification head of the CNN, and the CNN further includes a segmentation head configured to predict, for a given pixel in the image data, whether the pixel represents a particular embryo feature).
Regarding claim 4, Hall et al. discloses the system uses a standard loss function for segmentation models (either BianryCrossEntropy or standard Crossentropy loss, both of which are segmentation loss functions). Hall et al. further discloses that a loss function may be defined to asses performing of a model, and during training the deep learning model is optimized by varying learning rates to drive the update mechanism for the network's weight parameters to minimize an objective/loss function. Hall et al. further discloses that in some embodiments the loss function is modified to incorporate distribution effects (which includes using relative weights). These may include cross-entropy (CE) loss, weighted CE, residual CE, inference distribution or a custom loss function (para. 0091, 00108, 00131; the system of claim 3, wherein the CNN is trained using a loss function that includes a classification loss for the classification head, and a segmentation loss for the segmentation head, and the loss function includes a relative weight of the classification loss and segmentation loss).
Pertaining to claim 5, Hall et al. discloses that the AI model configured to generate an embryo viability score identifies Zona Pel lucida regions (para. 0007; the system of claim 3, wherein the particular of embryo feature is an inner cell mass, a trophectoderm, or a zona).
Pertaining to claim 6, Hall et al. discloses that stops and other components may be used to restrict or modify illumination to certain parts of the image (or image plane) (para. 0072; the system of claim 1, wherein the polarized light image data includes a frame reflecting a particular imaged layer of the mammal embryo specimen).
Regarding claim 7, Gu et al. discloses that the first mitotic spindles were imaged at least five times from different directions by rotating samples at an equatorial plane of each embryo to ensure the accuracy of three-dimensional information of mitotic spindles (p. 748, col. 1, para. 2; the system of claim 1, wherein the polarized light image data includes a plurality of frames, each reflecting a particular imaged layer of the mammal embryo specimen).
Pertaining to claim 10, Hall et al. teaches receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilisation (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label (e.g. heart beat detected at first scan post IVF) (para. 0007, 0063; the system of claim 1, wherein the instructions, when executed by the processor cause the processor to: provide metadata of the mammal embryo specimen to the CNN).
Pertaining to claim 13, Hall et al. discloses that in some embodiments deployment comprises saving or exporting the trained model, such as by writing the model weights and associated model metadata to a file. Hall et al. further disclose that deployment may comprise exporting the model coefficients and model metadata to a file. It would be obvious to try storing these weights and metadata in table format (para. 0068, 00210; the system of claim 10, wherein the instructions, when executed by the processor cause the processor to: maintain a look-up table for mapping values of the metadata to values trained with the CNN).
Pertaining to claim 14, Hall et al. discloses that demographic information includes patient age and Hall et al. further discloses a study of the effect of patient age on the accuracy of the ensemble-based Al model was conducted (para. 00147; 00155; 00181; the system of claim 10, wherein the metadata includes a patient's age).
Pertaining to claim 15, Hall et al. teaches that the images analyzed are of human embryos (para. 0039; the system claim 1, wherein the mammal is a human).
Regarding claim 16, Hall et al. teaches that images are pre-processed and that this can comprise cropping the image using deep learning or computer vision method, and this can comprise padding the image, normalizing the colour balance, normalizing the brightness and scaling the image to a predefined resolution. Hall et al. further teaches that during training of an Al model one or more augmented images are generated for each image in the training set and during assessment of the validation set, the results for the one or more augmented images are combined to generate a single result for the image (para. 0016-0020; the system of claim 1, wherein the instructions, when executed by the processor cause the processor to generate the polarized light image data).
9. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Hall et al. (WO2020198779A1; 7/12/2023 IDS document), in view of Tilia et al. (Fertility and Sterility, 2016, vol 105, p. 1085-2000) and Gu et al. (reproductive BioMedicine Online, Vol. 19, 2009, p. 745-754), as evidenced by Golubovsky (Human Reproduction, 2003, Vol. 18, p. 236-242), as applied to claims 1-7, 10 and 13-17 above, and further in view of Chen et al. (AVEC ’19, 2019, Cross-cultural Emotion Sub-challenge Session, Nice, France, p. 19-26). The italicized text corresponds to the instant claim limitations.
The limitations of claims 1-7, 10 and 13-17 were taught by Hall et al., Tilia et al. and Gu et al. above.
Pertaining to claim 8, Hall et al., Tilia et al. and Gu et al. are silent to: the system of claim 7, wherein the CNN includes an inner layer configured to produce a plurality of representation vectors, each corresponding to one of the plurality of frames, and the representation vectors are provided to a 1D convolutional layer of the CNN. However, these limitations were known in the art at the time of the effective filing date of the invention as taught by Chen et al.
Regarding claim 8, Chen et al. teaches dimension reduction of spatial-temporal features for CNN analysis with 2D-1D architecture wherein each image frame is processed by Resnet50 network to yield a frame-based representation. Chen et al. further discloses that the representations are stacked and fed into a 1D convolutional network. Under broadest reasonable interpretation, the output of the ResNet feature extractor is the output of an inner CNN layer (abstract, Fig. 1; the system of claim 7, wherein the CNN includes an inner layer configured to produce a plurality of representation vectors, each corresponding to one of the plurality of frames, and the representation vectors are provided to a 1D convolutional layer of the CNN).
An invention would have been prima facie obvious to one of ordinary skill in the art at the effective filing date of the invention if some motivation in the prior art would have led that person to combine the prior art teachings to arrive at the claimed invention. Chen et al. taught that applying their approach to reduce features in multi-frame image analysis by using frame-based representation followed by 1D convolution network, improved emotional state recognition by outperforming baseline performance (abstract, Fig. 1, p. 25, col. 2, para. 2 – p. 26, col. 1, para. 1). Therefore, one of ordinary skill in the art would have been motivated to utilize the method reduce features in multi-frame image analysis taught by Chen et al. in the method and system for selecting viable embryos taught by Hall et al., in order to reduce features while improving performance. Furthermore, one of ordinary skill in the art would predict that the feature reduction/representation taught by Chen et al. could be readily added to the system of Hall et al. with a reasonable expectation of success because they both pertain to CNN analysis of multi-frame images. The invention is therefore prima facie obvious.
10. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Hall et al. (WO2020198779A1; 7/12/2023 IDS document), in view of Tilia et al. (Fertility and Sterility, 2016, vol 105, p. 1085-2000) and Gu et al. (reproductive BioMedicine Online, Vol. 19, 2009, p. 745-754), as evidenced by Golubovsky (Human Reproduction, 2003, Vol. 18, p. 236-242), as applied to claims 1-7, 10 and 13-17 above, and further in view of Tran et al. (2015 IEEE International Conference on Computer Vision,IEEE Computer society, 2015, p. 4489 - 4494). The italicized text corresponds to the instant claim limitations.
The limitations of claims 1-7, 10 and 13-17 are taught by Hall et al., Tilia et al. and Gu et al. above.
The limitations of claim 8 are taught by Hall et al., Tilia et al., Gu et al. and Chen et al. above.
Regarding claim 9, Hall et al., Tilia et al., and Gu et al. are silent to: the system of claim 7, wherein the CNN is a 3D convolutional neural network and the polarized image data is organized as a volume including the plurality of frames. However, this limitation was known in the art at the time of the effective filing date of the invention as taught by Tran et al.
Pertaining to claim 9, Tran et al. discloses applying a 3D convolution neural network on a video volume resulting in another volume, preserving temporal information of the input signal. (Fig. 1c; p. 4490, col. 2, para. 2; the system of claim 7, wherein the CNN is a 3D convolutional neural network and the polarized image data is organized as a volume including the plurality of frames).
An invention would have been prima facie obvious to one of ordinary skill in the art at the effective filing date of the invention if some motivation in the prior art would have led that person to combine the prior art teachings to arrive at the claimed invention. Tran et al. taught that when applied to a large-scale supervised video dataset, compared to 2D convolution and 2D pooling operations, their model that performs 3D convolutions and 3D pooling is simple and effective and more suitable for spatiotemporal feature learning compared to 2D ConvNets (p. 4490, col. 2, para. 2; abstract). Therefore, one of ordinary skill in the art would have been motivated to apply the 3-dimensional convolutional networks taught by Tran et al. in the method and system for selecting viable embryos taught by Hall et al., Tilia et al. and Gu et al. in order to learn spatiotemporal features compared to 2D convolution. Furthermore, one of ordinary skill in the art would predict that the 3D convolution networks taught by Tran et al. could be readily added to the system of Hall et al., Tilia et al. and Gu et al. with a reasonable expectation of success because they both pertain to CNN analysis of multi-frame images. The invention is therefore prima facie obvious.
11. Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Hall et al. (WO2020198779A1; 7/12/2023 IDS document), in view of Tilia et al. (Fertility and Sterility, 2016, vol 105, p. 1085-2000) and Gu et al. (reproductive BioMedicine Online, Vol. 19, 2009, p. 745-754), as evidenced by Golubovsky (Human Reproduction, 2003, Vol. 18, p. 236-242), as applied to claims 1-7, 10 and 13-17 above, and further in view of Li et al. (2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020, p. 1996-2000). The italicized text corresponds to the instant claim limitations.
The limitations of claims 1-7, 10 and 13-17 are taught by Hall et al., Tilia et al. and Gu et al. above.
Regarding claims 11 and 12, Hall et al. and Tilia et al. and Gu et al. are silent to: the system of claim 10, wherein the metadata is provided to an inner layer of the CNN (claim 11); and the system of claim 11, wherein the metadata is concatenated to the output of a layer preceding the inner layer (claim 12). However, these limitations were known in the art at the time of the effective filing date of the invention as taught by Li et al.
Regarding claim 11, Li et al. teaches a method of fusing metadata and dermoscopy images for skin disease diagnosis by deep learning analysis. Li et al. further teaches a method of integrating image data with metadata by multiplication-based data fusion and is fed into to a two-layer fully connected sub-network of the CNN (abstract, Fig. 1b; p. 1997, col. 1, para. 2; the system of claim 10, wherein the metadata is provided to an inner layer of the CNN).
With respect to claim 12, Li et al. further teaches a second method of integrating image data with metadata by conventional concatenation-based data fusion. By this method, the metadata information is represented by a one-dimensional vector, and is directly concatenated with the visual feature vector extracted from the last layer of a CNN, followed by one or more fully connected layers (Fig. 1a; p. 1997, col. 1, para. 1; the system of claim 11, wherein the metadata is concatenated to the output of a layer preceding the inner layer).
An invention would have been prima facie obvious to one of ordinary skill in the art at the effective filing date of the invention if some motivation in the prior art would have led that person to combine the prior art teachings to arrive at the claimed invention. Li et al. taught that using both non-image patient metadata and image data for intelligent diagnosis of skin disease to include more data in diagnosis improves performance, especially for rare disease categories (abstract, p. 1999, col. 2, para. 2). Therefore, one of ordinary skill in the art would have been motivated to utilize the data fusion approaches taught by Li et al. in the method and system for selecting viable embryos taught by Hall et al., Tilia et al. and Gu et al in order to use more data other than image data in classification to improve performance, especially for rare classes. Furthermore, one of ordinary skill in the art would predict that the data fusion methods taught by Li et al. could be readily added to the system of Hall et al., Tilia et al. and Gu et al. with a reasonable expectation of success because they both pertain to CNN analysis to classify images in applications where non-image-based patient data is also available. The invention is therefore prima facie obvious.
12. Claims 18, 20-21 and 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Molinari et al. (J. Assist Reprod. Genet, 2012, Vol. 29, p. 1117-1122; 12 July 2023 IDS), in view of Gu et al, as evidenced by Golubovsky (Human Reproduction, 2003, Vol. 18, p. 236-242). The italicized text corresponds to the instant claim limitations.
Pertaining to claim 18, Molinari et al. discloses collecting images using a polarization microscope (Oosight, CRI, USA) and that the Ooosight Meta™ software coupled with automatic ZP and MS detector was used to acquire data for both oocytes and embryos. Molinari et al. further discloses that all PLM
images acquired on oocytes and embryos were recorded and later analyzed. The use of software to acquire image data inherently requires a computer and memory to execute the software and store the software and the data (p. 1119, col. 1, para. 1-3; p. 1119, col. 2, para. 1; Fig. 2; a computer-implemented system comprising: an image sensor; a processor in communication with the image sensor and a memory in communication with the processor; the memory storing instructions that, when executed by the processor cause the processor to [perform the method]).
Regarding claim 18, Molinari et al. discloses that the Oosight Meta™ software coupled with automatic zona pellucida and meiotic spindle detector was used to acquire data for both oocytes and embryos. Tilia et al. discloses the automated detection by Oosight™ software of birefringent structures of oocytes and embryos. Molinari et al. further discloses that a bright signal corresponding to oocyte MS was automatically located and highlighted in red, and that data are obtained regarding retardance, length of the major axis and area. Molinari et al. further discloses the automatic detection of the area and retardance of the inner layer of the zona pellucida (IL-ZP), that is particularly birefringent (p. 1119, col. 1, para. 3 – col. 2, para.1; Fig. 2; receive, from the image sensor, image data representing emerging polarized light that has traversed a specimen).
Pertaining to claim 18, Molinari et al. discloses that the image analysis was accomplished using the tools provided by the software: the “automatic-detection” button specifically identifies the boundaries of birefringent signals, thus determining the area of the birefringent structure, usually a barrel-shaped red area for the MS and a ring-like red structure for the IL-ZP. Molinari et al. further discloses that the software provided direct bidimensional measurements of the area of both meiotic spindle and zona pellucida structures. (Fig. 2; p. 1119, col. 2, para. 1-2; determine birefringence properties of the specimen based at least in part on the image data).
Pertaining to claim 18, Molinari et al. discloses that after calibration was set, oocytes and embryos were analyzed at 400× magnification and gently rotated in order to visualize the meiotic spindle and the zona pellucida in the brightest focus plane. Acquired images were analyzed using software tools that analyzed birefringent signals including providing direct bidimensional measurements of meiotic spindle and ZP structures (p. 1119, col. 1,para. 2 – col. 2, para. 2; generate a polarized light image representative of the specimen based at least in part on the birefringence properties).
Pertaining to claim 18 Molinari et al. discloses that features of the polarized light images were analyzed to determine if features correlate with a successful clinical pregnancy (i.e. if a clinical pregnancy was obtained in 34 cycles (conception cycles ; CCs) or if 52 cycles were unsuccessful (non-conception cycles; NCCs). The features of the polarized light that were analyzed for significant correlation with CC versus NCC were: meiotic spindle (MS) area, MS major axis length, MS retardance (in oocytes) and inner layer of the zona pellucida (IL-ZP) area and IL-ZP retardance in oocytes and embryos (Table 2; p. 1120, col. 1, para. 3 – p. 1121, col. 1, para. 2; p. 1090, col. 1, para. 1; classify features of the polarized light image using a classifier).
Pertaining to claims 18, 20 and 21, Molinari et al. is silent to the limitation directed to: identify features of the polarized light image as mitotic spindles; determine mitotic activity of the specimen based at least in part on the identified mitotic spindles; predict a ploidy status of the specimen based on the mitotic activity (claim 18); the system of claim 18, wherein the mitotic activity of the specimen is determined based at least in part on a number of the identified mitotic spindles (claim 20); and the system of claim 18, wherein the memory stores further instructions that, when executed by the processor cause the processor to determine geometric shapes of the identified mitotic spindles; and the mitotic activity of the specimen is determined based at least in part on the geometric shapes of the identified mitotic spindles (claim 21). However, these limitations were known in the art at the time of the effective filing date of the invention as taught by Gu et al.
Pertaining to claim 18, Gu et al. teaches identifying mitotic spindles in human 3PN zygotes using polarized light by Polscope analysis, which relies on birefringence (an inherent physical property of highly ordered molecules such as microtubules) (p. 749, col. 1, para. 3 – col. 2, para. 2; Fig. 2; p. 746, col. 2, para. 3; identify features of the polarized light image as mitotic spindles).
Regarding claim 18, Gu et al. teaches observing the mitotic spindle morphology in human tripronuclear (3PN) zygotes using polarized light images. Gu et al. discloses that 3PN zygotes with bipolar mitotic spindles were observed to cleave to 2-cell embryos and those with tripolar spindles to cleave to 3-cell embryos, demonstrating that abnormal mitotic spindles resulted in abnormal cleavage patterns, which inherently indicates mitotic activity (abstract; p. 751, col. 2, para. 3; determine mitotic activity of the specimen based at least in part on the identified mitotic spindles; ).
Regarding claim 18, to make predictions about the relationship of spindle morphology in 3PN zygotes to abnormal ploidy status of the resulting embryo, Gu et al. uses ploidy data from several prior studies summarized in Gu et al. and Golubovsky et al. and described below: It is known that the 3 variants of human 3PN zygote cleavage correspond to 3 outcomes: (i) mitotic division with biopolar spindle gives 3PN blastomeres and embryos (triploid) in 25% of cases, (ii) exclusion of one haploid genome from the metaphase plate of the first cleavage division (14–32% of cases), results in 2PN diploid, 2PN/3PN mosaics (diploid and triploid) and 1PN/2PN diploid derivatives; and (iii) in 50–60% of 3PN zygotes, a tripolar spindle is formed at the first cleavage division, resulting in dramatic abnormalities in chromosome distribution, resulting in aneuploidy.
Therefore, based on these disclosed data, zygotes with tripolar spindles that Gu et al. observed to cleave to 3-cell embryos, inherently correspond to aneuploidy in the embryo and zygotes with dipolar spindles that Gu et al. observed to cleave to 2-cell embryos, inherently correspond to heteroploidy or tripoloidy in the embryo. Gu et al. further predicts the ploidy status of abnormal mitotic spindles: In their experiment wherein the extra pronucleus is removed in 3PN zygotes, they predict that: 1) it is the bipolar mitotic spindle during the first cleavage that produces diploid embryos and 2) the percentage of 2n cells in the 2n/3n mixploid embryos (in outcome (ii) above) is increased, and diploidization increases potential to reach blastocyst stage (Gu et al. p. 750, col. 2, para. 2 – p. 753, col. 1, para. 1; Golubovsky Figure 1; predict a ploidy status of the specimen based on the mitotic activity).
Pertaining to claim 20, Gu et al. discloses comparing mitotic spindles in two groups of 3PN zygotes, those with and without removal of the extra pronucleus, by polscope analysis based on the number of zygotes with tripolar spindles, multiple spindles bipolar spindles or irregular spindles. Gu et al. further teaches using these counts to make conclusions about mitotic activity. Specifically, Gu et al. found that in the first cleavage, the human 3PN zygotes with bipolar mitotic spindles were seen to cleave to 2-cell embryos and human 3PN zygotes with typical tripolar mitotic spindles cleaved to 3-cell embryos in both groups and concluded that these observations proved that abnormal mitotic spindles
resulted in abnormal cleavage patterns (p. 749, col. 2, para. 3 – p. 750, col. 2, para. 1; Table 3; p. 751, col. 2, para. 3; the system of claim 18, wherein the mitotic activity of the specimen is determined based at least in part on a number of the identified mitotic spindles).
Pertaining to claim 21, Gu et al. discloses measuring shape of the spindles and that the shape reflects tripolar versus bipolar. Gu et al. discloses that a polarized light microscope was used to continually observe the first cleavage of human 3PN zygotes and found that 3PN zygotes with bipolar mitotic spindles were directly observed to cleave to 2-cell embryos and 3PN zygotes with tripolar spindles cleaved to 3-cell embryos. Gu et al. further discloses that bipolar and tripolar mitotic spindles had different shapes (p. 751, col. 2; para. 3; Fig. 2; the system of claim 18, wherein the memory stores further instructions that, when executed by the processor cause the processor to determine geometric shapes of the identified mitotic spindles; and the mitotic activity of the specimen is determined based at least in part on the geometric shapes of the identified mitotic spindles).
An invention would have been prima facie obvious to one of ordinary skill in the art at the effective filing date of the invention if some motivation in the prior art would have led that person to combine the prior art teachings to arrive at the claimed invention. Gu et al. taught that the polarized light microscopy provides a rapid, simplified and non-invasive method for identifying mitotic spindles in human embryos and that mitotic spindle abnormalities may be a major pathway leading to post zygotic chromosomal abnormalities, which may account for the developmental loss in human embryos (p. 753, col. 2, para. 1). Therefore, one of ordinary skill in the art would have been motivated to utilize the method to detect mitotic spindle abnormality from polarized light images taught by Gao et al. in the computer-based system for analyzing and classifying samples using birefringence features in polarized light images taught by Molinari et al. in order to use mitotic spindle morphology in predicting chromosome abnormalities in human embryos. Furthermore, one of ordinary skill in the art would predict that the mitotic spindle analysis methods taught by Gu et al. could be readily added to the system of Molinari et al. with a reasonable expectation of success because they both pertain to generation and analysis of polarized light images of embryos for implantation based on birefringence features. The invention is therefore prima facie obvious.
Regarding claim 23, Molinari et al. disclose that in their system, human embryos are analyzed by polarized light microscopy (abstract; the system of claims 18, wherein the specimen is from a mammal embryo).
Pertaining to claim 24, Molinari et al. disclose that in their system, human embryos are analyzed by polarized light microscopy (abstract; the system of claim 23, wherein the mammal is a human).
Conclusion
13. No claims are allowed.
Claims 19 and 22 were examined for patentability under U.S.C. 102 and U.S.C. 103 and were found to be free from the prior art. Regarding claim 19, there were no prior studies alone or in combination found that disclose using polarized light image analysis to determine that when the mitotic activity is below the predetermined threshold the ploidy status of the specimen is predicted to be euploid (i.e. having the correct normal number of chromosomes). Regarding claim 22, there are no prior studies that alone or in combination that teach using polarized light images to identify an inner cell mass (ICM; a.k.a. embryoblast) and a trophectoderm (TE; a.k.a. trophoblast) in an embryo, and determine locations of the identified mitotic spindles as in the ICM or in the TE. No studies show measurement of ICM or TE using polarized light microscopy because they are not ordered birefringent structures, and thus are difficult to detect.
Claims 1-17 were examined for patent eligibility under U.S.C. 101 and were found to be patent eligible at Step 2B because additional element limitations in claims 1 and 17 were found to be not well-understood, routine and conventional. These limitations in claim 1 are those directed to “receiving polarized light image data reflective of a mammal embryo specimen” and “present the polarized light image data to a convolutional neural network (CNN) trained to classify specimens according to a ploidy status”, and the limitations of claim 17 are “receiving polarized light image data reflective of a mammal embryo specimen” and “presenting the polarized light image data to a convolutional neural network (CNN) trained to classify according to a ploidy status”. Claims 2-16 were also considered to be patent eligible by virtue of their dependence on claim 1.
E-mail Communications Authorization
14. Per updated USPTO Internet usage policies, Applicant and/or applicant's representative is encouraged to authorize the USPTO examiner to discuss any subject matter concerning the above application via Internet e-mail communications. See MPEP 502.03. To approve such communications, Applicant must provide written authorization for e-mail communication by submitting the following statement via EFS-Web (using PTO/SB/439) or Central Fax (571-273-8300): "Recognizing that Internet communications are not secure, / hereby authorize the USPTO to communicate with the undersigned and practitioners in accordance with 37 CFR 1.33 and 37 CFR 1.34 concerning any subject matter of this application by video conferencing, instant messaging, or electronic mail. / understand that a copy of these communications will be made of record in the application file."
Written authorizations submitted to the Examiner via e-mail are NOT proper. Written authorizations must be submitted via EFS-Web (using PTO/SB/439) or Central Fax (571-273- 8300). A paper copy of e-mail correspondence will be placed in the patent application when appropriate. E-mails from the USPTO are for the sole use of the intended recipient, and may contain information subject to the confidentiality requirement set forth in 35 USC § 122. See also MPEP 502.03.
Inquiries
15. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNIFER J SMITH whose telephone number is (571)272-7801. The examiner can normally be reached Monday-Friday 7:00 AM - 3:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Olivia Wise can be reached at (571) 272-2249. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.J.S./Examiner, Art Unit 1685
/OLIVIA M. WISE/Supervisory Patent Examiner, Art Unit 1685
Prosecution Insights