DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-5 and 9-10 are amended. Claims 1-11 are pending in this application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al., US 2019/0188446 in view of Cooke et al., “Physics-enhanced machine learning for virtual fluorescence microscopy”.
Regarding claim 1, Wu discloses a computer-implemented method for predicting digital fluorescence images (fig. 2; para 0005 and 0030; a computer-implemented method for generating virtually stained images (i.e., digital fluorescence images) of unstained samples), the method comprising
capturing a first digital image of a tissue sample (fig. 2, element 220; para 0026, 0028 and 0034; the imaging system 135 may be a microscope may be used to acquire the images that are stored in the image sample data storage 140. An image of the unstained second tissue sample may be accessed. The image includes a plurality of spectral images of the unstained second tissue sample. For example, the image may be accessed from the image sample data storage 140 or the image sample data storage 141) by means of a microsurgical optical system (figs. 3(a)-3(b); para 0039; a microscope) with a first digital image capturing unit (figs. 3(a)-3(b), element 310; para 0040; a camera) with a first plurality of color channel information using white light (figs. 3(a)-3(b), element 342; para 0041; white light) and at least one optical filter (figs. 3(a)-3(b), elements 331 and 332; para 0039-0040; an optical filters),
predicting a second digital image in a form of a digital fluorescence representation of the captured first digital image by means of a trained machine learning system comprising a trained learning model that outputs a predicted digital fluorescence representation of an input image (fig. 2, elements 225-230; para 0034; the trained artificial neural network 130 may be used to generate a virtually stained image of a second unstained tissue sample; The virtually stained image may then be output),
wherein the captured first digital image is used as input image for the trained machine learning system (fig. 2, element 220; para 0034; An image of the unstained second tissue sample may be accessed. The image includes a plurality of spectral images of the unstained second tissue sample (i.e., the image is inputted into the trained artificial neural network)), and
wherein parameter values of one or more parameters of the at least one optical filter were determined during training of the machine learning system (fig. 2, element 210; para 0006, 0008, and 0031; accessing a set of parameters for an artificial neural network. The set of parameters includes weights associated with artificial neurons within the artificial neural network; An output layer of the artificial neural network may include three artificial neurons that respectively predict red, blue, and green channels (i.e., parameter values of the optical filter) of the virtually stained image).
Wu discloses claim 1 as enumerated above, but Wu does not explicitly disclose the joint optimization including simultaneously training of the machine learning system and optimizing the one or more parameters of the at least one optical filter, and the parameter values being used in predicting the second digital image as claimed.
However, Cooke discloses determining physical imaging system parameters during training of a machine learning model through joint optimization. Specifically, Cooke teaches that a physical model of the microscope in included in the learning network and is jointly optimized during DNN training. Cooke further discloses that the supervised training process optimizes both the physical parameters of our hardware (LED brightness and color values) and the parameters of our DNN. Thus, Cooke teaches simultaneously training the machine learning mode and optimizing imaging system parameters used for subsequent fluorescence image prediction (Abstract; Section: I. Introduction, Section: II., A. Image Formation, Second paragraph, and Section: V. Results and Discussion, C. LED Parameterization, Fifth paragraph).
Therefore, taking the combined disclosures of Wu and Cooke as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the disclosure of Cooke into the invention of Wu for the benefit allowing the joint optimization of illumination and image processing we achieve consistently better performance than all tested alternatives (Cooke: Section VI. Conclusion, Third paragraph).
Regarding claim 2, the method according to claim 1, Wu in the combination further disclose wherein training the learning model of the trained machine learning system comprises:
providing a plurality of first digital training images of tissue samples (fig. 2, element 205; para 0028, 0030, and 0037-0038; the imaging system 135 may be a microscope may be operated in various modes in order to acquire different images of a sample. For example, the imaging system 135 may be used to acquire the images 120 and the images 125; The image training dataset includes a plurality of image pairs, each of which includes a first image 120 of an unstained first tissue sample and a second image 125 of the first tissue sample after staining), which were captured under white light (figs. 3(a)-3(b), element 342; para 0041; white light) by means of a microsurgical optical system (figs. 3(a)-3(b); para 0039; a microscope) with a second image capturing unit (figs. 3(a)-3(b), element 310; para 0040; a camera), wherein a second plurality of color channel information for different spectral ranges are available for each first digital training image,
providing a plurality of second digital training images each representing the same tissue samples as the first set of digital training images, wherein the second digital training images have indications of diseased elements of the tissue samples (fig. 2, element 205; para 0028, 0030, 0034, and 0037-0038; the imaging system 135 may be a microscope may be operated in various modes in order to acquire different images of a sample. For example, the imaging system 135 may be used to acquire the images 120 and the images 125; The image training dataset includes a plurality of image pairs, each of which includes a first image 120 of an unstained first tissue sample and a second image 125 of the first tissue sample after staining; The second tissue sample may include the same tissue type as the first tissue sample that was used to train the artificial neural network 130…such as whether the tissue is healthy or diseased with various types of disease and/or severity of disease),
training the machine learning system for forming the trained machine learning model for predicting a digital image of a type of the plurality of second digital training images, wherein use is made of the following as input values for the machine learning system (fig. 2, element 215; para 0032; The artificial neural network 130 may then be trained by using the image training data set and the parameter set to adjust some or all of the parameters associated with the artificial neurons within the artificial neural network 130, including the weights within the parameter set):
the plurality of first digital training images in the form of the second plurality of color channel information (fig. 2, element 210; para 0006, 0008, and 0031; accessing a set of parameters for an artificial neural network. The set of parameters includes weights associated with artificial neurons within the artificial neural network; An output layer of the artificial neural network may include three artificial neurons that respectively predict red, blue, and green of the virtually stained image),
the plurality of second digital training images as ground truth (fig. 2, element 215; para 0032-0033; The artificial neural network 130 may then be trained by using the image training data set and the parameter set (i.e., ground truth)),
parameter values for reducing the second plurality of color channel information by means of at least one digitally simulated optical filter for forming the first plurality of color channel information (fig. 2, element 210; para 0006, 0008, and 0031; accessing a set of parameters for an artificial neural network. The set of parameters includes weights associated with artificial neurons within the artificial neural network; An output layer of the artificial neural network may include three artificial neurons that respectively predict red, blue, and green channels (i.e., parameter values of the optical filter) of the virtually stained image),
wherein the plurality of first digital training images is used as training data for predicting a digital image of the type of the plurality of second digital training images after the second plurality of color channel information has been reduced to the first plurality of color channel information by means of the digitally simulated optical filter (fig. 2, element 215; para 0032; The artificial neural network 130 may then be trained by using the image training data set and the parameter set to adjust some or all of the parameters associated with the artificial neurons within the artificial neural network 130, including the weights within the parameter set. For example, the weights may be adjusted to reduce or minimize a loss function of the artificial neural network 130), and
wherein at least one portion of the parameter values of the one or more parameters of the at least one optical filter is output as output values of the machine learning system after the training of the machine learning system has ended (fig. 2, element 210; para 0006, 0008, and 0031; accessing a set of parameters for an artificial neural network. The set of parameters includes weights associated with artificial neurons within the artificial neural network; An output layer of the artificial neural network may include three artificial neurons that respectively predict red, blue, and green channels (i.e., parameter values of the optical filter) of the virtually stained image).
Regarding claim 3, the method according to claim 1, Wu in the combination further disclose wherein the parameter values of the one or more parameters of the at least one optical filter comprise: the plurality of first color channel information and/or a filter shape of the digitally simulated optical filter (para 0006, 0008, 0031, and 0039-0040).
Regarding claim 4, the method according to claim 2, Wu in the combination further disclose wherein the second plurality of color channel information is greater than the first plurality of color channel information (para 0036-0038).
Regarding claim 5, the method according to claim 2, Wu in the combination further disclose wherein the parameter values for reducing the second number of color channels are at least one selected from the group consisting of a filter shape and a respective central frequency of the first plurality of color channel information (fig. 2, element 210; para 0006, 0008, and 0031).
Regarding claim 6, the method according to claim 2, Wu in the combination further disclose wherein parameter values for controlling the source of the white light during the capturing of the first digital image are generated as additional output values of the machine learning system after the training of the machine learning system has ended (fig. 2, element 210; para 0006, 0008, and 0031).
Regarding claim 7, the method according to claim 1, Wu in the combination further disclose wherein the digital fluorescence representation corresponds to a representation such as would be generated using a light source in the UV range (figs. 3(a)-3(b), element 301; para 0039).
Regarding claim 8, the method according to claim 1, Wu in the combination further disclose wherein the learning model corresponds to an encoder-decoder model in terms of its set-up (para 0031 and 0033; U-NET and/or Convolutional Neural Network).
Regarding claim 9, the method according to claim 8, Wu in the combination further disclose wherein the encoder-decoder model is a convolutional network in the form of a U-Net architecture (para 0031 and 0033).
Regarding claim 10, this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons.
Regarding claim 11, this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons.
Response to Arguments
Applicant's arguments with respect to claims 1-11 have been considered but are moot in view of the new ground(s) of rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN D HUYNH whose telephone number is (571)270-1937. The examiner can normally be reached 8AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAN D HUYNH/Primary Examiner, Art Unit 2665
Prosecution Insights