DETAILED ACTION
The applicant’s response, from 15 May 2026, has been fully considered. Amendments to the claims, from 15 May 2026, were received and entered. The following rejections and/or objections are either reiterated or newly applied. They constitute the complete set presently being applied to the instant application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 8 and 9 are cancelled.
Claims 1-7 and 10-13 are currently pending and under examination herein.
Claims 1-7 and 10-13 are rejected.
Priority
Foreign priority is acknowledged to EP21184360.2 filed 07/07/2021. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. In this action, claims 1-7 and 10-13 are examined as though they had an effective filing date of 07/07/2021. In future actions, the effective filing date of one or more claims may change, due to amendments to the claims, or further analysis of the disclosures of the priority applications.
Information Disclosure Statement
The information disclosure statements (IDS) filed on 07/06/2022 and 07/20/2022 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Drawings
The drawings filed on 07/06/2022 are accepted.
Claim Rejections - 35 USC § 112
The previously issued 35 USC 112(b) rejection is withdrawn in response to the amendments to the claims.
Claim Rejections - 35 USC § 101
The previously issued 35 USC 101 rejection is withdrawn in response to the applicant’s arguments (see response to arguments below). The following rejection is newly made against the amended claims (the action is non-final).
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, and 10-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. In accordance with MPEP 2106, claims found to recite statutory subject matter (Step 1: YES) are then analyzed to determine if the claims recite any concepts that equate to an abstract idea or natural law (Step 2A, Prong 1). Claims 1-7 and 10 are directed to methods and Claims 11-13 are directed to systems. In the instant application, the claims recite the following limitations that equate to an abstract idea:
Claim 1 recite the limitation - extracting features from the microscope image having information on a state of the biological system; using the features and the metadata to predict the future state of the biological system; identifying a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state; and generating data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation. Based on the broadest reasonable interpretation, extracting features, making a prediction, identifying a risk, and generating a command could practically be done by the human mind. This draws the limitation to a mental process, which classifies the limitation as an abstract idea.
Claim 2 recites the limitation - wherein the state of the biological system is related to at least one of a health, an activity, and a growth of the biological system. This limitation specifies the state used in the identifying judicial exception of claim 1. The refined judicial exception indicated by this limitation still represents a judicial expectation.
Claim 3 recites the limitation - wherein the metadata comprises information on at least one of a configuration of a microscope, used for generating the microscope image, an environmental condition, an agent interacting with the biological system at the associated time, a temperature, a pH, a partial pressure of carbon dioxide, a partial pressure of oxygen, a humidity, a culture condition of the biological system, a type or amount of a buffer solution, nutrient, antibiotic or growth factor of the biological system at the associated time. This limitation specifies the metadata used in the predicting and identifying judicial exceptions of claim 1. The refined judicial exceptions indicated by this limitation still represents a judicial expectation.
Claim 6 recites extracting further features from the further microscope image having information on the state of the biological system; and using the features and the further features and the metadata and the further metadata to predict the future state by detecting anomalies based on a temporal development between the features and the further features and between the metadata and further metadata. Based on the broadest reasonable interpretation, extracting features and making a prediction could practically be done by the human mind. This draws the limitation to a mental process, which classifies the limitation as an abstract idea.
Claim 7 recites the limitation - using the features and the metadata to predict the future state of the biological system. Based on the broadest reasonable interpretation, making a prediction could practically be done by the human mind. This draws the limitation to a mental process, which classifies the limitation as an abstract idea.
Claim 10 recites the limitation - wherein the risk parameter relates to an illumination property, a temperature, a humidity, an oxygen level, a carbon dioxide level or an agent having an influence on the state of the biological system. This limitation specifies the parameter used in the identifying and generating judicial exceptions of claim 1. The refined judicial exceptions indicated by this limitation still represents a judicial expectation.
Claim 11 recites the limitation - extract features from the microscope image having information on a state of the biological system; use the features and the metadata to predict the future state of the biological system; identify a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state; and generate data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation. Based on the broadest reasonable interpretation, extracting features, making a prediction, identifying a risk, and generating a command could practically be done by the human mind. This draws the limitation to a mental process, which classifies the limitation as an abstract idea.
Claim 12 recites the limitation - predict a future state of the biological system. This is reciting the predating judicial exception of claim 11. The recited judicial exception still represents a judicial exception.
Claim 13 recites the limitation - performing the method according to claim 1. Claim 1 recites judicial exceptions (see claim 1 judicial exceptions above).
These limitations recite concepts of identifying, predicting, and determining information that are so generically recited that they can be practically performed in the human mind as claimed, which falls under the “Mental processes” grouping of abstract ideas. MPEP 2106.04 indicates a claim that requires a computer may still recite a mental process. The claims indicate nothing to imply that the identifying, predicting, and determining information could not be performed mentally. Therefore, these limitations fall under the “Mental process” grouping of abstract ideas. As such, claims 1-7 and 10-13 recite an abstract idea (Step 2A, Prong 1: YES).
Claims found to recite a judicial exception under Step 2A, Prong 1 are then further analyzed to determine if the claims as a whole integrate the recited judicial exception into a practical application or not (Step 2A, Prong 2). These judicial exceptions are not integrated into a practical application because the claims do not recite an additional element that reflects an improvement to technology (i.e. the computer/computing or microscope/camera/image generation) (MPEP 2106.04(d)(1)) or a particular treatment and prophylaxis (MPEP 2106.04(d)(2)). Rather, the claims provide insignificant extra-solution activity (MPEP 2106.05(g)) and provide mere instructions to apply a judicial exception (MPEP 2106.05(f)). Specifically, the claims recite the following additional elements:
Claim 1 recites receiving a microscope image depicting the biological system at an associated time; receiving metadata corresponding to the microscope image.
Claim 4 recites extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture.
Claim 5 recites using the features and the metadata comprises detecting anomalies by means of a second trained artificial neural network, the further second trained artificial neural network being trained based on a sequence of microscope images, depicting biological systems over time, and a corresponding sequence of metadata over the time.
Claim 6 recites receiving a further microscope image depicting the biological system at another associated time; receiving further metadata corresponding to the further microscope image;
Claim 7 recites receiving a microscope image depicting the biological system at an associated time; receiving metadata corresponding to the microscope image; extracting features from the microscope image having information on a state of the biological system, wherein extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture; using a decoder of the trained artificial neural network having the encoder-decoder architecture to reconstruct a segmented image based on the future state being predicted, the segmented image depicting the biological system as one or more segments according to the future state.
Claim 11 recites processor and a memory storing instructions; receive a microscope image depicting a biological system at an associated time; receive metadata corresponding to the microscope image.
Claim 12 recites a microscope configured to generate a microscope image depicting a biological system at an associated time; processor and a memory storing instructions; and receive the microscope image.
Claim 13 recites computer-readable medium comprising a program code.
There are no limitations that indicate that the claimed identifying, predicting, and determining information require anything other than generic computing systems. As such, these limitations equate to mere instructions to implement the abstract idea on a generic computer that the courts have stated does not render an abstract idea eligible. There is no indication that these steps are affected by the judicial exception in any way and thus do not integrate the recited judicial exception into a practical application. As such, claims 1-7 and 10-13 are directed to an abstract idea (Step 2A, Prong 2: NO).
Claims found to be directed to a judicial exception are then further evaluated to determine if the claims recite an inventive concept that provides significantly more than the judicial exception itself (Step 2B). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite conventional additional elements that equate to mere instructions to apply the recited exception in a generic way or in a generic computing environment. The claims also recite conventional additional elements that represent insignificant extra-solution activities.
As discussed above, there are no additional limitations to indicate that the claimed identifying, predicting, and determining information require anything other than generic computer components in order to carry out the recited abstract idea in the claims. Claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea or natural law eligible. MPEP 2106.05(f) discloses that mere instructions to apply the judicial exception cannot provide an inventive concept to the claims. As specified in MPEP 2106.05(g), extra-solution activities can be understood as incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Insignificant extra-solution activities include mere data gathering, selecting a particular data source or type of data to be manipulated, and displaying information. Additionally, Moen et al. (2019, Nature Methods, Vol.16: 1233-1446) teach capturing images with microscopes, receiving images, using neural networks for image analysis, making predictions from image analysis, and computers were well understood, routine, and conventional at the time of the effective filing date (Page 1234, Column 1, Paragraph 2: We then review four use cases: image classification, image segmentation, object tracking, and augmented microscopy; Page 1234, Column 2, Paragraph 2: Once training data have been acquired, a deep learning model can be trained to accurately make predictions for new data; Page 1243, Column 1, Paragraph 1: Attention must be paid to metadata).
The additional elements do not comprise an inventive concept when considered individually or as an ordered combination that transforms the claimed judicial exception into a patent-eligible application of the judicial exception. Therefore, the claims do not amount to significantly more than the judicial exception itself (Step 2B: No). As such, Claims 1-7 and 10-13 are not patent eligible.
Response to arguments
The applicant argues that the claims do not recites a law of nature (Page 10, Paragraph 2). The applicant asserts that the claims are not directed to any naturally occurring relationship; rather, the risk parameter is a parameter of user controllable metadata-for example, an illumination property, temperature, humidity, oxygen level, or carbon-dioxide level. The examiner agrees that the recited claims are not drawn to a law nature through a natural correlation based on the guidance of MPEP 2106.04(b). Therefore, the previously issued rejection is withdrawn. A new 35 USC 101 rejection has been made against the amended claims, without including law of nature judicial exceptions (See above).
Claim Rejections - 35 USC § 103
The previously issued 35 USC 102 and 103 rejections are withdrawn in response to the applicant’s arguments (see response to arguments below). The following 35 USC103 rejection is newly made against the amended claims (the action is non-final).
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over Haan et al. (2020, npj digital medicine, Vol. 3: 1-9, Cited in previous Office Action), in view of Zhang et al. (2020, IEEE Transactions on Medical Imaging, Vol. 39, No. 4: 1-13). Italicized text from reference art.
The applicable claims include:
Claim 1. A method for predicting a future state of a biological system, comprising: (Claim 1.i) receiving a microscope image depicting the biological system at an associated time; (Claim 1.ii) receiving metadata corresponding to the microscope image; (Claim 1.iii) extracting features from the microscope image having information on a state of the biological system; (Claim 1.iv) using the features and the metadata to predict the future state of the biological system; (Claim 1.v)identifying a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state; and (Claim 1.vi) generating data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation.
Claim 2. The method according to claim 1, wherein the state of the biological system is related to at least one of a health, an activity, and a growth of the biological system.
Claim 3. The method according to claim 1, wherein the metadata comprises information on at least one of a configuration of a microscope, used for generating the microscope image, an environmental condition, an agent interacting with the biological system at the associated time, a temperature, a pH, a partial pressure of carbon dioxide, a partial pressure of oxygen, a humidity, a culture condition of the biological system, a type or amount of a buffer solution, nutrient, antibiotic or growth factor of the biological system at the associated time.
Claim 4. The method according to claim 1, wherein extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture.
Claim 5. The method according to claim 1, wherein using the features and the metadata comprises detecting anomalies by means of a second trained artificial neural network, the further second trained artificial neural network being trained based on a sequence of microscope images, depicting biological systems over time, and a corresponding sequence of metadata over the time.
Claim 6. The method according to claim 1, further comprising: (Claim 6.i) receiving a further microscope image depicting the biological system at another associated time;(Claim 6.ii) receiving further metadata corresponding to the further microscope image; (Claim 6.iii) extracting further features from the further microscope image having information on the state of the biological system; and (Claim 6.iv) using the features and the further features and the metadata and the further metadata to predict the future state by detecting anomalies based on a temporal development between the features and the further features and between the metadata and further metadata.
Claim 7. A method for predicting a future state of a biological system, comprising: (Claim 7.i) receiving a microscope image depicting the biological system at an associated time; (Claim 7.ii) receiving metadata corresponding to the microscope image; (Claim 7.iii) extracting features from the microscope image having information on a state of the biological system, (Claim 7.iv) wherein extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture; (Claim 7.v) using the features and the metadata to predict the future state of the biological system; and (Claim 7.vi) using a decoder of the trained artificial neural network having the encoder-decoder architecture to reconstruct a segmented image based on the future state being predicted, the segmented image depicting the biological system as one or more segments according to the future state.
Claim 10. The method according to claim 1, wherein the risk parameter relates to an illumination property, a temperature, a humidity, an oxygen level, a carbon dioxide level or an agent having an influence on the state of the biological system.
Claim 11. An apparatus for predicting a future state of a biological system, comprising a processor and a memory storing instructions that, when executed by the processor, cause the apparatus to: (Claim 11.i) receive a microscope image depicting a biological system at an associated time; (Claim 11.ii) receive metadata corresponding to the microscope image; (Claim 11.iii) extract features from the microscope image having information on a state of the biological system; (Claim 11.iv) use the features and the metadata to predict the future state of the biological system; (Claim 11.v) identify a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state; and (Claim 11.vi) generate data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation.
Claim 12. A system, comprising: (Claim 12.i) a microscope configured to generate a microscope image depicting a biological system at an associated time; and (Claim 12.ii) an apparatus for predicting a future state of a biological system according to claim 11, the apparatus comprising a processor and a memory storing instructions that, when executed by the processor, cause the apparatus to receive the microscope image to predict a future state of the biological system.
Claim 13. A non-transitory, computer-readable medium comprising a program code for performing the method according to claim 1 when the program code is executed by a processor.
Regarding Claim 1, 11, and 13, Haan et al. teach (Claim 1.i) receiving a microscope image depicting the biological system at an associated time (Page 5, Column 2, Paragraph 1: We used thin blood smear slides for image analysis. Our ground truth microscope images were obtained using a scanning benchtop microscope). The use of the images within the modeling indicates they were received. Haan et al. teach (Claim 1.ii) receiving metadata corresponding to the microscope image (Page 5, Column 2, Paragraph 4: The co-registration between the smartphone microscope images and those taken by the clinical benchtop microscope). The data to maintain the connection is interpreted as metadata corresponding to the image. All data other than the images themselves are interpreted as metadata. Haan et al. teach (Claim 1.iii) extracting features from the microscope image having information on a state of the biological system (Page 7, Column 1, Paragraph 2: A second deep neural network is used to perform semantic segmentation of the blood cells imaged by our smartphone microscope). The state of the system is whether the blood cells are currently healthy or not (i.e. their shape). The extracted features are the cells. Haan et al. teach (Claim 1.iv) using the features and the metadata to predict the future state of the biological system (Page 7, Column 1, Paragraph 1: At the end of this whole process, which is a one-time training effort, three classes are defined for the subsequent semantic segmentation training of the neural network: (1) sickle, (2) normal red blood cell, and (3) background). The output of the Semantic segmentation model is a class label describing if a cell shape is indicative of sickle cell. Sickle cell deformation is due to a genetic mutation (Page 1, Column 1, Paragraph 2). Therefore the deformed characteristics are permanent. Because of this, a cell that is deformed due to sick cell disease will be deformed in the future (i.e. the classification of the image represents a prediction of the future state). The disclosure does not indicate a limiting definition of future state that must be different from the current state for the system (i.e. the shape of the cell). Additionally, Haan et al. teach the methods utilize a computer that inherently has non-transitory computer readable media, memory, and at least one processor (Page 7, Column 2, Paragraph 5: The networks were trained and test images were processed on a desktop computer). Claim 11 and 13 recites the limitations of claim 1 directed to systems.
Regarding Claim 2, Haan et al. teach the state of the biological system is related to at least one of a health, an activity, and a growth of the biological system (Page 1, Column 1, Paragraph 2: As a result of this, the normally biconcave disc-shaped red blood cells become crescent or sickle-shaped in people living with SCD; Page 7, Column 1, Paragraph 2: A second deep neural network is used to perform semantic segmentation of the blood cells imaged by our smartphone microscope). The state of the system (cell health) is whether the blood cells are currently healthy or not (i.e. their shape).
Regarding Claim 3, Haan et al. teach the metadata comprises information on at least one of a configuration of a microscope, used for generating the microscope image (Page 5, Column 2, Paragraph 4: The co-registration between the smartphone microscope images and those taken by the clinical benchtop microscope) The data to maintain the connection is interpreted as metadata corresponding to the images.
Regarding Claim 4, Haan et al. teach extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture (Page 7, Column 1, Paragraph 2: A second deep neural network is used to perform semantic segmentation of the blood cells imaged by our smartphone microscope. This network has the same architecture as the first image enhancement network (U-net)). The U-net architecture uses an encoder of an encoder decoder architecture. This is a trained neural network (Page 6, Column 2: Mask creation for training the cell segmentation network).
Regarding Claim 5, Haan et al. teach detecting anomalies by means of a second trained artificial neural network, the second trained artificial neural network being trained based on a sequence of microscope images, depicting biological systems over time, and a corresponding sequence of metadata over the time (Page 6, Figure 5: Diagram detailing the network architecture for both a the image enhancement network and b the semantic segmentation network). Haan et al. utilizes two neural networks as part of their methods, each utilizes series of images. Detecting anomalies is equivalent to identifying the sickle cell red blood cells. Additionally, both Haan et al. and Zhang et al. teach detecting anomalies by means of a trained artificial neural network indicating that there are two.
Regarding Claim 7, Haan et al. teach (Claim 7.i) receiving a microscope image depicting the biological system at an associated time (Page 5, Column 2, Paragraph 1: We used thin blood smear slides for image analysis. Our ground truth microscope images were obtained using a scanning benchtop microscope). The use of the images within the modeling indicates they were received. Haan et al. teach (Claim 7.ii) receiving metadata corresponding to the microscope image (Page 5, Column 2, Paragraph 4: The co-registration between the smartphone microscope images and those taken by the clinical benchtop microscope) The data to maintain the connection is interpreted as metadata corresponding to the images. All data other than the images themselves are interpreted as metadata. Haan et al. teach (Claim 7.iii) extracting features from the microscope image having information on a state of the biological system (Page 7, Column 1, Paragraph 2: A second deep neural network is used to perform semantic segmentation of the blood cells imaged by our smartphone microscope). The state of the system is whether the blood cells are currently healthy or not (i.e. their shape). The extracted features are the cells. Haan et al. teach (Claim 7.iv) wherein extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture (Page 7, Column 1, Paragraph 2: A second deep neural network is used to perform semantic segmentation of the blood cells imaged by our smartphone microscope. This network has the same architecture as the first image enhancement network (U-net)). The U-net architecture uses an encoder of an encoder decoder architecture. This is a trained neural network (Page 6, Column 2: Mask creation for training the cell segmentation network). Haan et al. teach (Claim 7.v) using the features and the metadata to predict the future state of the biological system (Page 7, Column 1, Paragraph 1: At the end of this whole process, which is a one-time training effort, three classes are defined for the subsequent semantic segmentation training of the neural network: (1) sickle, (2) normal red blood cell, and (3) background). The output of the Semantic segmentation model is a class label describing if a cell shape is indicative of sickle cell. Sickle cell deformation is due to a genetic mutation (Page 1, Column 1, Paragraph 2). Therefore the deformed characteristics are permanent. Because of this, a cell that is deformed due to sick cell disease will be deformed in the future (i.e. the classification of the image represents a prediction of the future state). The disclosure does not indicate a limiting definition of future state that must be different from the current state for the system (i.e. the shape of the cell). Haan et al. suggests (Claim 7.vi) using a decoder of the trained artificial neural network having the encoder-decoder architecture to reconstruct a segmented image based on the future state being predicted, the segmented image depicting the biological system as one or more segments according to the future state (Page 6, Figure 5: b the semantic segmentation network). The decoder of the trained segmentation network of Hann et al. produces a segmented image showing cells classified as sickle cell. This is an obvious representation of a future state of the system (see Haan et al. teaching of Claim 1.iv).
Regarding Claim 12, Haan et al. teach (Claim 12.i) a microscope configured to generate a microscope image depicting a biological system at an associated time (Page 5, Column 1, Paragraph 5: Design of the smartphone-based brightfield microscope We used a Nokia Lumia 1020 smartphone attached to a custom-designed 3D-printed unit to capture images of the blood smear slides). Haan et al. and Zhang et al. teach (Claim 12.ii) the apparatus of claim 11 (see regarding claims 1, 11 and 13).
Haan et al. does not teach identifying a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state (Claim 1.v). Haan et al. does not teach generating data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation (Claim 1.vi). Haan et al. does not teach using a decoder of the trained artificial neural network having the encoder-decoder architecture to reconstruct a segmented image based on the future state being predicted, the segmented image depicting the biological system as one or more segments according to the future state (Claim 7.vi).
Regarding Claim 1, 11, and 13, Zhang et al. teach (Claim 1.ii) receiving metadata corresponding to the image (Page 2, Column 2, Paragraph 1: A novel spatio-temporal Convolutional Long Short-Term Memory (ST-ConvLSTM) network is proposed to jointly learn the intra-slice spatial structures, the inter-slice correlations in 3D contexts, and the temporal dynamics in time sequences; Page 4, Column 1, Paragraph 1: Three image feature channels are derived: 1) intracellular volume fraction (ICVF) images representing the cell density that is normalized between, 2) post-contrast CT images in soft-tissue window, 3) binary tumor segmentation mask; Page 5, Column 2, Paragraph 3: Along with the image features, clinical factors have non-neglectful influences on predicting the future image as well. We integrate the related factors into our model). Metadata is received and utilized within the modeling to discriminate 2 dimensional info (x,y), 3 dimensional info (z), and 4 dimensional data (time) from the photos, in addition to the image feature (pre vs post contrast agent) and clinical feature metadata indicated. All data other than the images themselves are interpreted as metadata. Metadata was received to differentiate pre and post contrast photos. Photos were also tied to specific patients. Zhang et al. teach (Claim 1.iii) extracting features from the image having information on a state of the biological system (Page 3, Column 2, Paragraph 3: for each pair of pre- and post-contrast 3D CT volumes at the same time point, their organ regions are first roughly cropped and registered to post-contrast CT. The segmentation is performed manually; Page 4, Column 1, Paragraph 1: Three image feature channels are derived: 1) intracellular volume fraction (ICVF) images representing the cell density that is normalized between, 2) post-contrast CT images in soft-tissue window, 3) binary tumor segmentation mask). The state of the system (i.e. state of the patient) is the information related to the pancreatic tumors, and includes image segmentation and data generated from the image related to the tumors. Zhang et al. teach (Claim 1.iv) using the features and the metadata to predict the future state of the biological system (Page 5, Column 1, Paragraph 2: Along with the current input image, the ST-CLSTM unit can predict the future slice; Page 5, Column 2, Paragraph 3: the ST-ConvLSTM determines the future state by jointly considering or integrating the compact spatial information of the current slice, the states of slices from previous times and adjacent locations, and clinically relevant factor(s). After that, the decoder with four deconvolutional layers generates the future frame). Zhang et al. teach (Claim 1.v) identifying a risk parameter of the metadata, the risk parameter having a positive correlation with a degradation of the state of the biological system with respect to the future state (Page 7, Column 1, Paragraph 2: our new ST-ConvLSTM model is holistically 4D (volumetric+time) image-based and enables the predictions of future tumor imaging properties, such as future cell density and CT intensity numbers to assist relevant clinical diagnosis). Risk parameters (parameters of tumors such as growth rate, size, cell density) are identified for the metadata of the photos and clinical factors positively correlated with the predicted future tumor growth (i.e. degradation of the patient). Zhang et al. teach (Claim 1.vi) generating data for an external entity having an influence on the risk parameter, the data comprises a command for the external entity to adapt a configuration related to the risk parameter for mitigating the degradation (Page 1, Column 2, Paragraph 1: the patient-specific prediction of PanNET’s growth pattern at earlier stages is highly desirable, since it will assist decision making on different treatment strategies to better manage the undergoing treatment or surgical planning; Page 7, Column 1, Paragraph 2: our new ST-ConvLSTM model is holistically 4D (volumetric+time) image-based and enables the predictions of future tumor imaging properties, such as future cell density and CT intensity numbers to assist relevant clinical diagnosis). It is obvious that the risk parameter generated from the metadata represent a command for doctors related to tumor monitoring, surgery to remove the tumor, or other treatments based on specific thresholds presented by the art (Page 1, Column 2, Paragraph 1: treatments of pancreatic neuroendocrine tumor (PanNET or PNET) include active surveillance, surgical intervention, and medical treatment. Active surveillance is undertaken if a PanNET does not reach 3 cm in diameter or a tumor-doubling time <500 days; otherwise the corresponding PanNET should be resected due to the high risk of metastatic disease). Additionally, Zhang et al. teach the methods utilize a computer that inherently has non-transitory computer readable media, memory, and at least one processor (Page 12, Column 2, Paragraph 3: The authors thank Nvidia for the TITAN X Pascal GPUs).
Regarding Claim 2, Zhang et al. teach the state of the biological system is related to at least one of a health, an activity, and a growth of the biological system (Page 3, Column 2, Paragraph 3: for each pair of pre- and post-contrast 3D CT volumes at the same time point, their organ regions are first roughly cropped and registered to post-contrast CT). The state of the system (patient health) is the information related to the pancreatic tumors (tumor progression/size).
Regarding Claim 3, Zhang et al. teach the metadata comprises information on at least one an agent interacting with the biological system at the associated time (Page 2, Column 2, Paragraph 1: A novel spatio-temporal Convolutional Long Short-Term Memory (ST-ConvLSTM) network is proposed to jointly learn the intra-slice spatial structures, the inter-slice correlations in 3D contexts, and the temporal dynamics in time sequences; Page 4, Column 1, Paragraph 1: Three image feature channels are derived: 1) intracellular volume fraction (ICVF) images representing the cell density that is normalized between, 2) post-contrast CT images in soft-tissue window, 3) binary tumor segmentation mask; Page 5, Column 2, Paragraph 3: Along with the image features, clinical factors have non-neglectful influences on predicting the future image as well. We integrate the related factors into our model). Metadata is received and utilized within the modeling to discriminate 2 dimensional info (x,y), 3 dimensional info (z), and 4 dimensional data (time) from the photos, in addition to the image feature (pre vs post contrast agent) and clinical feature metadata indicated. The cancer was an agent. No limiting definition of agent was found within the specification, so the plain meaning of the term was used - something that produces or is capable of producing an effect.
Regarding Claim 4, Zhang et al. teach extracting features from the image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture (Page 5, Column 2, Paragraph 3: Specifically, each frame in the 4D spatio-temporal space is recurrently passed into the encoder which consists of four convolutional layers to encode a feature map).
Regarding Claim 6, Zhang et al. teach (Claim 6.i) receiving a further image depicting the biological system at another associated time (Page 3, Column 2, Paragraph 3: Our 4D longitudinal tumor imaging data set used in this study consists of dual-phase contrast-enhanced CT volumes at three time points for each patient). Zhang et al. teach (Claim 6.ii) receiving further metadata corresponding to the further microscope image (Page 4, Column 1, Paragraph 1: The dataset is prepared for every tumor volume at each time point, and imaging volumes at different times are aligned using the segmented 3D tumor centroids, to build the spatio-temporal sequence data set for training and testing). The metadata indicated in for claim 1 above is collected for each time point. Zhang et al. teach (Claim 6.iii) extracting further features from the further microscope image having information on the state of the biological system (Page 5, Column 1, Paragraph 2: Instead of simply concatenating the 2D CT slices, in order to learn simultaneously both the spatial consistency patterns among successive image slices and the temporal dynamics across different time points, we propose a new Spatio-Temporal Convolutional LSTM (ST-ConvLSTM) network as illustrated in Fig. 2). The modeling is run for each time point (there were multiple for each patient). Time was an important variable considered by the study. Zhang et al. teach (Claim 6.iv) using the features and the further features and the metadata and the further metadata to predict the future state by detecting anomalies based on a temporal development between the features and the further features and between the metadata and further metadata. (Page 5, Column 1, Paragraph 2: Instead of simply concatenating the 2D CT slices, in order to learn simultaneously both the spatial consistency patterns among successive image slices and the temporal dynamics across different time points, we propose a new Spatio-Temporal Convolutional LSTM (ST-ConvLSTM) network as illustrated in Fig. 2). The modeling is run for each time point (there were multiple for each patient). Time was an important variable considered by the study. No limiting definition for anomalies was found in the specification.
Regarding Claim 7, Zhang et al. teach (Claim 7.ii) receiving metadata corresponding to the microscope image (Page 2, Column 2, Paragraph 1: A novel spatio-temporal Convolutional Long Short-Term Memory (ST-ConvLSTM) network is proposed to jointly learn the intra-slice spatial structures, the inter-slice correlations in 3D contexts, and the temporal dynamics in time sequences; Page 4, Column 1, Paragraph 1: Three image feature channels are derived: 1) intracellular volume fraction (ICVF) images representing the cell density that is normalized between, 2) post-contrast CT images in soft-tissue window, 3) binary tumor segmentation mask; Page 5, Column 2, Paragraph 3: Along with the image features, clinical factors have non-neglectful influences on predicting the future image as well. We integrate the related factors into our model). Metadata is received and utilized within the modeling to discriminate 2 dimensional info (x,y), 3 dimensional info (z), and 4 dimensional data (time) from the photos, in addition to the image feature (pre vs post contrast agent) and clinical feature metadata indicated. All data other than the images themselves are interpreted as metadata. Zhang et al. teach (Claim 7.iii) extracting features from the microscope image having information on a state of the biological system (Page 3, Column 2, Paragraph 3: for each pair of pre- and post-contrast 3D CT volumes at the same time point, their organ regions are first roughly cropped and registered to post-contrast CT. The segmentation is performed manually; Page 4, Column 1, Paragraph 1: Three image feature channels are derived: 1) intracellular volume fraction (ICVF) images representing the cell density that is normalized between, 2) post-contrast CT images in soft-tissue window, 3) binary tumor segmentation mask). The state of the system is the information related to the pancreatic tumors. Zhang et al. teach (Claim 7.iv) wherein extracting features from the microscope image comprises using an encoder of a trained artificial neural network having an encoder-decoder architecture (Page 5, Column 2, Paragraph 3: Specifically, each frame in the 4D spatio-temporal space is recurrently passed into the encoder which consists of four convolutional layers to encode a feature map). Zhang et al. teach (Claim 7.v) using the features and the metadata to predict the future state of the biological system (Page 5, Column 1, Paragraph 2: Along with the current input image, the ST-CLSTM unit can predict the future slice; Page 5, Column 2, Paragraph 3: the ST-ConvLSTM determines the future state by jointly considering or integrating the compact spatial information of the current slice, the states of slices from previous times and adjacent locations, and clinically relevant factor(s). After that, the decoder with four deconvolutional layers generates the future frame). Zhang et al. teach (Claim 7.vi) using a decoder of the trained artificial neural network having the encoder-decoder architecture to reconstruct a segmented image based on the future state being predicted, the segmented image depicting the biological system as one or more segments according to the future state (Page 5, Column 2, Paragraph 3: After that, the decoder with four deconvolutional layers generates the future frame).
Regarding Claim 10, Zhang et al. teach the risk parameter relates to an agent having an influence on the state of the biological system (Page 7, Column 2, Paragraph 2: We calculate the scatter plots of the ST-ConvLSTM predicted tumor volumes and the respective growth rates; Page 11, Column 1, Paragraph 1: it enables the prediction of both future images and the associated imaging properties, including CT scan, tumor cell density and radiodensity, as demonstrated in this paper). The cancer was considered an agent and all parameters associated with the cancers are associated with a risk parameter. No limiting definition of agent was found within the specification, so the plain meaning of the term was used - something that produces or is capable of producing an effect.
Regarding Claim 12, Haan et al. and Zhang et al. each teach (Claim 12.ii) the apparatus of claim 11 (see regarding claims 1, 11 and 13).
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to combine the methods of Zhang et al. with Haan et al. Haan et al. teaches cost effective, versatile, and quick methods for prediction information about cells related to a disease using images and a neural network (Page 2, Column 1, Paragraph 3: Here we present a smartphone-based microscope and machine learning algorithms that together form a cost-effective, portable, and rapid sickle cell screening framework, facilitating early diagnosis of SCD even in resource-limited settings). Relatedly, Zhang et al. teach an effective and efficient methods for prediction information about organs related to a disease using images and a neural network (Page 2, Column 2, Paragraph 1: We demonstrate the effectiveness and high efficiency of employing 4D ST-ConvLSTM for a 3D+time left-ventricle ultrasound image segmentation task. Only a small subset of sparsely-annotated 3D ultrasound volumes per time sequence are required by ST-ConvLSTM). Therefore, it would have been obvious to someone of ordinary skill in the art at the time of the effective filling date to combine the methods from the references indicated above. Furthermore, one of ordinary skill in the art would predict that the methods could be readily combined with a reasonable expectation of success because both are within the same technical field – making predictions related disease using images and neural networks. Additionally, the model architecture of both references are related, employing encoders and decoders to process images, facilitating their combination. Accordingly, Claims 1-7 and 10-13 taken as a whole would have been prima facie obvious before the effective filing date and are rejected under 35 U.S.C. 103.
Response to Arguments
The applicant argues that Vicar et al. does not disclose a command as required by amended claims 1 and 11 (previously claim 9) (Page 14, Paragraph 4). The applicant asserts Vicar et al. discloses no command output of any kind directed to any external entity but is framed as descriptive characterization, not predictive control. The examiner agrees that the results of Vicar et al. do not constitute a command as the art sets forth no motivation to act on the generated information related to cell death. Therefore, the previously issued rejection is withdrawn. A new 35 USC 103 rejection has been made against the amended claims, that includes a motivation to interpret the data generated as a command (See above).
Double Patenting
No double patenting is identified.
Conclusion
No claims are allowed.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAKE H ELKINS whose telephone number is (571)272-2649. The examiner can normally be reached Monday-Thursday 8-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Karlheinz Skowronek can be reached at (571) 272-9047. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.H.E./Examiner, Art Unit 1687
/Karlheinz R. Skowronek/Supervisory Patent Examiner, Art Unit 1687
Prosecution Insights