DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 18863333, filed on 11/06/2024.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/06/2024, 01/18/2025, 10/27/2025, and 04/09/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “computing device” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “process and analyze time-lapse images of the microorganisms and/or colonies thereof”. In this limitation, “time-lapse images” has improper antecedence and should recite “the time-lapse images”. It is being interpreted accordingly.
Claims 2-15 are rejected as dependent on claim 1.
Claims 4 and 13 recite, “detect candidate microorganisms and/or colonies thereof”, which has improper antecedence and should recite “detect the candidate microorganisms and/or colonies thereof”. It is being interpreted accordingly.
Claim 13 recites the limitation "the time-lapse holographic images". There is insufficient antecedent basis for this limitation in the claim. This is being interpreted as a new element.
Claim 14 is rejected as dependent on claim 13.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 6 and 10 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 6 describes the computing device as local and/or remote, which covers any computing device, thus claim 6 does not limit claim 1, on which it depends. Claim 10 describes the TFT-based sensor as disposable, which does not limit the TFT-based sensor of claim 1. Under broadest reasonable interpretation, any TFT-based sensor can be disposed of (i.e. is disposable). Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of detecting microorganisms in time-lapse images, without significantly more.
The claim recites: “A system for the detection and classification of live microorganism and/or colonies thereof in a sample using time-lapse imaging comprising:
a light source;
a thin film transistor (TFT)-based image sensor located along an optical path originating from the light source;
a growth plate containing growth medium thereon and containing the sample interposed along the optical path and disposed adjacent to the TFT-based image sensor;
a microcontroller or other circuitry configured to periodically illuminate the growth plate with light from the light source and capture time-lapse images of microorganisms and/or colonies thereof on the growth plate with the TFT-based image sensor; and
a computing device configured to execute image processing software to process and analyze time-lapse images of the microorganisms and/or colonies thereof on the growth plate and detect candidate microorganisms and/or colonies thereof in the time-lapse images.”
The limitations, as drafted, are processes that, under their broadest reasonable interpretation, cover performance of the limitation in the mind. A person can look at an image of microorganisms and/or colonies thereof, and detect the microorganisms and/or colonies. The time-lapse imaging amounts to insignificant, extra-solution activity (data collection).
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of a light source, a TFT-based image sensor, a growth plate with growth medium, a microcontroller, a computing device, and software. The light source, TFT-based sensor, and growth plate with growth medium amount to a generic microorganism imaging apparatus routinely used in applications such as digital holography. The microcontroller, computing device, and software amount to generic computational tools. Accordingly, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements are recited at a high-level of generality. It is therefore a judicial exception that is not integrated into a practical application, and does not include additional elements that are sufficient to amount to significantly more than the judicial exception. This claim is not patent eligible.
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of an incubator integrated with the imaging setup.
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of a light source with selected spectral bands (generic light source).
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to the mental processes of detecting candidate microorganisms and outputting a species classification. The claim also recites the additional elements of trained deep neural networks which are a recited at a high level of generality.
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of specifying various types of microorganisms.
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of a generically recited local or remote computing device.
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of adding a lens for magnification or de-magnification, which is a generic imaging component.
Claims 8-10 are rejected under 35 U.S.C. 101 because the claimed invention is directed to describing the generically recited TFT-based sensor without limiting it such that it is non-generic.
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of a chromogenic agar plates.
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a method corresponding to claim 1, particularly for the image acquisition (extra-solution activity, data collection).
Claims 13-14 are rejected under 35 U.S.C. 101 for containing limitations analogous to the above rejected claims 4 and 5. Claim 13 additionally recites differential image analysis, which amounts to a mental process as described in the claim.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an insignificant additional element of generic microbiology sample types.
Claims 16, 17, and 19 are rejected under 35 U.S.C. 101 for containing limitations analogous to the above rejected claims 1, 4, 5, and 7.
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to generic time-lapse image acquisition (extra-solution activity, data collection).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 5, 6, 10, 12, and 15 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Tanaka (JP6830593B2).
Regarding claim 1, Tanaka teaches “A system for the detection and classification of live microorganism and/or colonies thereof in a sample using time-lapse imaging comprising: a light source; a thin film transistor (TFT)-based image sensor located along an optical path originating from the light source; a growth plate containing growth medium thereon and containing the sample interposed along the optical path and disposed adjacent to the TFT-based image sensor; (Tanaka, Figure 1 (translated below, Paragraphs 15-17 and 41, “As shown in FIG. 1, the culture vessel 1 is placed on the photosensor 2, and the point light source 3 is arranged above the culture vessel 1. The culture vessel may be placed on the photosensor so that the colony forming image can be imaged, and as shown in FIG. 1, the culture container is placed so as to face the microbial culture surface (colony forming surface) with respect to the photosensor 2. It may be placed, or it may be placed with the colonization surface facing up. Here, the culture container is not particularly limited as long as it can cultivate microorganisms and can be loaded with the culture surface facing the photosensor, but for example, a frame seal (for example, a frame seal (for example) on a slide glass for microscope observation). A chamber formed by adhering several sheets (for example, two sheets) of in-situ PCR seals (for example, thickness: 300 μm, Frame-Seal Incubation Chambers, BIO RAD) covered with a release film and a cover film is preferable. Can be used. FIG. 2 shows the construction process of the chamber. The photosensor may be a sensor that can image an optical signal or a shadow signal derived from the formed microbial colony when the formed microbial colony is irradiated with light. For example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metall) is used. An image sensor such as an Oxford Sensor), a TFT (Thin Film Transistor) having a double gate structure, or a CIS Contact Image Sensor) can be used, but it is inexpensive and a CMOS sensor is preferably used because of its high resolution. As the CMOS sensor, it is desirable that the detection element size is 2 to 8 μm square, the number of elements is 300,000 to 18.1 million, and color imaging is possible. For example, 3.2 μm square detection elements are arranged in 2048 × 1536. However, DFK61BUC02 (Imaging Source, Germany) having an imaging area size of 6.55 mm × 4.92 mm can be mentioned.”; “Example 1 Construction of a lensless imaging system (1) An in-situ PCR sticker (thickness: 300 μm, Frame-Seal Innovation Chambers, BIO RAD) in which double-sided films are peeled off from a slide glass for microscope observation (thickness: 1.2 mm) ) Was pasted, and a second in-situ PCR sticker from which only the cover film was peeled off was pasted on top of it to prepare a 9 mm square, 600 μm high chamber (see FIG. 2). Pour 1.5% (w / v) low melting point agarose-added LB medium into the prepared chamber, place a cover glass (thickness: 170 μm) on it, flatten the surface of the agar medium, and leave it at room temperature for 20 minutes. The LB agar medium was solidified in. After peeling off the cover glass and the release film, 1 μl (5 × 10 4 cells / ml) of the bacterial cell suspension was dropped onto the agar medium, and the cover glass was placed again to perform plating.”)
PNG
media_image1.png
456
1019
media_image1.png
Greyscale
“a microcontroller or other circuitry configured to periodically illuminate the growth plate with light from the light source and capture time-lapse images of microorganisms and/or colonies thereof on the growth plate with the TFT-based image sensor;” (Tanaka, Paragraph 21, “The culture conditions may be appropriately set, for example, 18 hours at room temperature. The imaging may be performed at regular intervals during the culture time, for example, at intervals of 1 to 60 minutes, preferably at intervals of 5 minutes. The imaging conditions are not particularly limited as long as the characteristics of the colony forming image can be accurately extracted. For example, the exposure time is 1/100 to 1/4 second, preferably 1/18 to 1/10 second, and the frame rate is 3.75. It can be 10, preferably 3.75 to 5, and the white balance can be, for example, Red: 255, Green: 154, Blue: 64. Further, in order to reduce noise, it is preferable to continuously acquire a plurality of images and use the average image thereof.” Note that one skilled in the art would understand that a “microcontroller or other circuitry” would be required for the above limitation.)
“and a computing device configured to execute image processing software to process and analyze time-lapse images of the microorganisms and/or colonies thereof on the growth plate and detect candidate microorganisms and/or colonies thereof in the time-lapse images.” (Tanaka, Paragraphs 14 and 22, “The method for discriminating microorganisms of the present invention uses a lensless imaging system using a photosensor and analyzes an image (“lensless image”) obtained by imaging with the system. Here, the lensless imaging system means an optical sensing system that does not go through an objective lens. The lensless imaging system used in the method for discriminating microorganisms of the present invention includes a photosensor that acquires a colony formation image (scattered light pattern) of microorganisms as pixel data, a culture vessel for culturing a sample containing microorganisms, and microorganisms. It is composed of a light source that illuminates the colony and a computing device for extracting quantitative parameters from the pixel data of the colony formation image and performing multivariate analysis using them. A device including an incubator that controls the culture temperature as needed can be a device for carrying out the method for discriminating microorganisms of the present invention. FIG. 1 shows a configuration example of an apparatus for carrying out the method of the present invention.”; “Features extracted from a colony forming images, for example, the area, brightness, the shape and the like, colony maximum growth rate mu max, colony visualized time t a (Colony appearance time), the relative average intensity I, the deviation of the histogram G, donuts property D, At least the maximum colony growth rate μ max and the relative average brightness I selected from a total of 10 parameters of image entropy H, image energy density Ed , image energy E, weighted center difference W, and average brightness C in the central region. Three or more kinds including, or at least two kinds or more including deviation G and donut property D of the histogram, or donut property D and image entropy H are calculated.”)
Regarding claim 2, Tanaka teaches “The system of claim 1,”
“further comprising an incubator integrated with the light source, TFT-based image sensor, and growth plate.” (Tanaka, Paragraphs 14 and 22, “The method for discriminating microorganisms of the present invention uses a lensless imaging system using a photosensor and analyzes an image (“lensless image”) obtained by imaging with the system. Here, the lensless imaging system means an optical sensing system that does not go through an objective lens. The lensless imaging system used in the method for discriminating microorganisms of the present invention includes a photosensor that acquires a colony formation image (scattered light pattern) of microorganisms as pixel data, a culture vessel for culturing a sample containing microorganisms, and microorganisms. It is composed of a light source that illuminates the colony and a computing device for extracting quantitative parameters from the pixel data of the colony formation image and performing multivariate analysis using them. A device including an incubator that controls the culture temperature as needed can be a device for carrying out the method for discriminating microorganisms of the present invention. FIG. 1 shows a configuration example of an apparatus for carrying out the method of the present invention.”)
Regarding claim 3, Tanaka teaches “The system of claim 1,”
“wherein the light source comprises one or more selectively actuated spectral bands.” (Tanaka, Paragraph 18, “As the light source for illumination, an LED, an organic EL, a fluorescent lamp, an incandescent bulb, or the like can be used, but it is preferable to use an LED, and an LED having a wavelength in the range of 400 nm or more and 500 nm or less is used because the contrast of the colony image can be increased. Is more preferable.” Note that an LED emits light within a band of light.)
Regarding claim 5, Tanaka teaches “The system of claim 1,”
“wherein the microorganisms comprise a prokaryotic cell, a eukaryotic cell, bacteria, fungi, virus, multi-cellular organism, or clusters, films, or colonies thereof.” (Tanaka, Paragraphs 19-20, “In the method of the present invention, a sample containing a microorganism is cultured in a culture vessel placed on a photosensor array. Here, the "sample containing a microorganism" is a sample containing a microorganism to be discriminated and identified as a bacterial species, and may be either a clinical sample or a non-clinical sample. Clinical samples include, for example, blood, serum, plasma, blood fraction, joint fluid, urine, semen, saliva, feces, cerebrospinal fluid, gastric contents, vaginal secretions, tissue homogenate, bone marrow puncture, bone homogenate, sputum. , Suction fluid, swab and swab rinsate, other body fluids and the like. In addition, examples of non-clinical samples include substances including foods, beverages, pharmaceuticals, cosmetics, water, seawater ballasts, air, soil, sewage, plant materials, blood products, donor organs or tissue samples. In the present invention, the "microorganism" is not particularly limited as long as it forms a colony, and may be any of bacteria, fungi and the like. For example, Pseudomonas, Escherichia, Salmonella, Diarrhea, Enterobactor, Krebsiera, Seratia, Proteus, Camprovactor, Hemophilus, Morganella, Vibrio, Elsina, Asinetobacta, Stenotrophomonas. Genus, Brevendimonas, Larstonia, Achromobactor, Fuzobacterium, Prebotera, Blanchamera, Niseria, Burkholderia, Citrobacta, Hafnia, Edward Sierra, Aeromonas, Moraxera, Gram-negative bacteria such as Brucella, Pasturella, Providencia and Regionella; enterococcus, streptococcus, staphylococcus, bacillus, paenibacillus, lactic acid rod, listeria, peptstreptococcus, propionicate, crotridium Gram-positive bacteria such as Genus, Bacteroides, Gardnerella, Cochlear, Lactococcus, Leukonostock, Micrococcus, Mycobacteria and Corinebacterium; Candida, Cryptocox, Nocardia, Aokabi , Altanaria, Rhodotorula, Aspergillus, Fuzarium, Saccharomyces and Tricosporone.”)
Regarding claim 6, Tanaka teaches “The system of claim 1,”
“wherein the computing device comprises a local and/or remote computing device(s).” (Note that any computing device is either local and/or remote, thus the computing device of Tanaka inherently teaches this limitation.)
Regarding claim 10, Tanaka teaches “The system of claim 1,”
“wherein the TFT-based sensor is disposable.” (Under broadest reasonable interpretation, any object that can be disposed of is disposable, including a TFT-based sensor.)
Regarding claim 12, Claim 12 recites a method for using the system of claim 1. The rejection of claim 1 is applied here. Tanaka additionally teaches a method for the use of the system of claim 1 (Tanaka, Paragraph 14, “The method for discriminating microorganisms of the present invention uses a lensless imaging system using a photosensor and analyzes an image (“lensless image”) obtained by imaging with the system. Here, the lensless imaging system means an optical sensing system that does not go through an objective lens. The lensless imaging system used in the method for discriminating microorganisms of the present invention includes a photosensor that acquires a colony formation image (scattered light pattern) of microorganisms as pixel data, a culture vessel for culturing a sample containing microorganisms, and microorganisms. It is composed of a light source that illuminates the colony and a computing device for extracting quantitative parameters from the pixel data of the colony formation image and performing multivariate analysis using them. A device including an incubator that controls the culture temperature as needed can be a device for carrying out the method for discriminating microorganisms of the present invention. FIG. 1 shows a configuration example of an apparatus for carrying out the method of the present invention.”) Additionally, claim 12 specifies growth plate imaging at one or more spectral bands of illumination, which is disclosed by Tanaka as described in the above rejection of claim 3.
Regarding claim 15, Tanaka teaches “The system of claim 12,”
“wherein the sample comprises one or more of a water sample, a food sample, a biological or other fluid sample.” (Tanaka, Paragraphs 19-20, “In the method of the present invention, a sample containing a microorganism is cultured in a culture vessel placed on a photosensor array. Here, the "sample containing a microorganism" is a sample containing a microorganism to be discriminated and identified as a bacterial species, and may be either a clinical sample or a non-clinical sample. Clinical samples include, for example, blood, serum, plasma, blood fraction, joint fluid, urine, semen, saliva, feces, cerebrospinal fluid, gastric contents, vaginal secretions, tissue homogenate, bone marrow puncture, bone homogenate, sputum. , Suction fluid, swab and swab rinsate, other body fluids and the like. In addition, examples of non-clinical samples include substances including foods, beverages, pharmaceuticals, cosmetics, water, seawater ballasts, air, soil, sewage, plant materials, blood products, donor organs or tissue samples. In the present invention, the "microorganism" is not particularly limited as long as it forms a colony, and may be any of bacteria, fungi and the like. For example, Pseudomonas, Escherichia, Salmonella, Diarrhea, Enterobactor, Krebsiera, Seratia, Proteus, Camprovactor, Hemophilus, Morganella, Vibrio, Elsina, Asinetobacta, Stenotrophomonas. Genus, Brevendimonas, Larstonia, Achromobactor, Fuzobacterium, Prebotera, Blanchamera, Niseria, Burkholderia, Citrobacta, Hafnia, Edward Sierra, Aeromonas, Moraxera, Gram-negative bacteria such as Brucella, Pasturella, Providencia and Regionella; enterococcus, streptococcus, staphylococcus, bacillus, paenibacillus, lactic acid rod, listeria, peptstreptococcus, propionicate, crotridium Gram-positive bacteria such as Genus, Bacteroides, Gardnerella, Cochlear, Lactococcus, Leukonostock, Micrococcus, Mycobacteria and Corinebacterium; Candida, Cryptocox, Nocardia, Aokabi , Altanaria, Rhodotorula, Aspergillus, Fuzarium, Saccharomyces and Tricosporone.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Molaei (Imaging bacterial 3D motion using digital in-line holographic microscopy and correlation-based de-noising algorithm).
Regarding claim 7, Tanaka teaches “The system of claim 1,”
While Tanaka teaches a TFT-based sensor (see claim 1 rejection) used for hologram imaging of the microorganisms (one skilled in the art would understand the scattered light pattern generated by the imaging set-up of Figure 1 as hologram imaging; see Paragraphs 14 and 7), Tanaka does not expressly disclose the use of a lens to magnify or de-magnify these holograms onto the TFT-based sensor.
Molaei teaches using a magnification lens magnifying holograms of microorganisms onto photosensors (Molaei, Figure 1, and Section 3.2, “The DHM includes a CW (continuous wave) He-Ne laser, collimating optics, an inverted transmission microscope, and recording CCD camera ( Fig. 1 ). We illuminate the microchannel with a collimated laser beam generated by a 7mW He-Ne laser (Lamda equals 632.8 n m). The initial beam is filtered and collimated into a beam with the diameter of 5mm by a 20X objective (Edmund Scientific), a 25-um pinhole (Thorlabs), and a 25mm diameter doublet as the collimating lens f=50mm(, Newport Inc). A 1/20-λ aluminum mirror guides the horizontal beam downward into an inverted Nikon microscope (Nikon TS-100). To record bacteria holograms, an objective at the magnification of 40X (Nikon Super Plan Fluor ELWD, NA = 0.60) is used. The objective is focused on the plane 5 away from the bottom of the microfluidics. The holograms are recorded by a 2K × 2K CCD camera (Imperx 4M15L) with a pixel resolution of 0.185/pixel, which renders the lateral resolution of 0.2 for simplicity, and were streamed continuously at the rate of 15 fps to a data acquisition computer. The exposure time was 60 . To achieve robust estimations for each hologram in a series, the typical recording lasts about 20 minutes, totaling 18,000 holograms/acquisition.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate a lens for magnifying holograms onto a photosensor, as taught by Molaei, with respect to the TFT-based hologram imaging of microorganisms of Tanaka.
The motivation for doing so is described by Molaei (Molaei, Introduction, Paragraph 4, “Being an inherent 3-D recording technique and its ability in recording series of holograms digitally and reconstructing holograms numerically, digital holography paves the way for studying many dynamic phenomena [20]. However, the limited spatial resolution of the earlier digital holography systems [20] that are composed of lens-less recording cameras and the laser optics have proven difficult to observe micro-scale particles over substantial depth [20]. To circumvent the recording resolution limitation, Xu et al. has developed lens-less digital holography with a point illumination to visualize intra-cellular structure of a marine diatom [21]. Later, the technique has been implemented in a submersible to track marine particulates [22] with limited success. Using partial coherent illumination and multiple projections, Ozcan et al. [23–27] have developed several portable devices to screen and detect cells based on partial coherent holography. Amid numerous variations, their fundamental system consists of a large format sensor as the substrate, over which the cells are flowing, and a point source with partial coherence [28] that illuminates the shallow suspension from multiple angles. With the advantage of high resolution digital camera and strong near field scattering, they have successfully resolved cells with resolutions of ~1μm. However, the drawbacks of lens-less systems are the shallow sample depth (<20μm) with low concentration of sample cells (<105 cells/ml) and complicated post data analysis, since the magnification of the hologram in lens-less holography depends on axial distance of the object and the source of the reference beam from the hologram [20].”) Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Molaei to fully disclose “wherein a lens or set of lenses are used to magnify or de-magnify holograms of the microorganisms and/or colonies thereof onto the TFT-based image sensor.”
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Mudanyali (Lensless On-chip Imaging of Cells Provides a New Tool for High-throughput Cell-Biology and Medical Diagnostics).
Regarding claim 8, Tanaka teaches “The system of claim 1,”
Tanaka does not expressly disclose “wherein the TFT-based image sensor captures a field-of-view of at least 10 cm2.”
Mudanyali teaches sensors that capture a field-of-view of at least 10 cm2 (Mudanyali, Section A.1., “The LUCAS platform utilizes an optoelectronic sensor array to digitally record individual cell holograms. For this purpose, charged couple devices (CCD; Sample Models: KAI-11002, KAF-39000, from Kodak) or complementary metal-oxide-semiconductor chips (CMOS, Sample Model: MT9P031, Micron) can be used. Pixel sizes for the Kodak charged couple devices, KAI-11002, KAF-39000, and Micron CMOS image sensors are 9 μm, 6.8 μm and 2.2 μm, with an active FOV of 10 cm2, 18 cm2, and 24.4 mm2, respectively. [1-2].”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to increase the FOV of the TFT-base sensor of Tanaka to at least 10cm2, as taught by Mudanyali.
The motivation for doing so would have been to streamline imaging by capturing a larger area, thus removing the need to take multiple images of the same cell plate (or at least enabling the user to capture fewer images to image the full plate). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Mudanyali to fully disclose “wherein the TFT-based image sensor captures a field-of-view of at least 10 cm2.”
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Zheng (US20120223217A1).
Regarding claim 9, Tanaka teaches “The system of claim 1,”
Tanaka does not expressly disclose “wherein the TFT-based sensor is integrated on or within the growth plate.”
Zheng discloses sensor integration on or within a growth plate (Zheng, Figure 13 and Paragraph 186, “FIG. 13( a) is a photographic image of an e-Petri dish 620 according to an embodiment of the invention, and a quarter for size comparison. The e-Petri dish 620 includes a light detector 160, a transparent layer 165, and a well 170. The light detector 160 is in the form of a commercially available CMOS image sensor chip with a 6 mm×4 mm imaging area filled with 2.2 micron pixels. The microlens layer and color filter on the image sensor chip were removed to provide direct access to the image sensor pixels (light detecting elements 166). The microlens layer and color filter were removed by treating the sensor chip under oxygen plasma for 10 min (80 W). The transparent layer 165 in the form of a thin PDMS layer was prepared by mixing 1:10 with base and curing agent, then spin coated onto the sensing surface 162 followed by baking at 80° C. for 1 hour. The well 170 is a plastic square well comprising a peripheral wall 172 glued at the inner edges to the transparent layer 165 of the light detector 160 with poly-dimethylsiloxane (PDMS). The e-Petri dish 620 also includes a cover 176 hinged to an outer edge of the peripheral wall 172. In FIG. 13( a) a pipette is shown introducing a specimen 150 into the well 170.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to integrate the TFT-based sensor on the growth plate of Tanaka, based on the integration strategy taught by Zheng.
The motivation for doing so would have been to streamline experimental set-up. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Zheng to fully disclose “wherein the TFT-based sensor is integrated on or within the growth plate.”
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Meighan (US 20120149046 A1).
Regarding claim 11, Tanaka teaches “The system of claim 1,”
Tanaka does not expressly disclose “wherein the growth medium comprises chromogenic agar plates.”
Meighan teaches “wherein the growth medium comprises chromogenic agar plates.” (Meighan, Paragraph 3, “Chromogenic agars contain chromogenic substrates which are cleaved by enzymes produced by the target bacteria. Cleavage of the substrate changes the colour of the colony or surrounding agar and indicates that the target bacteria are present. Examples of chromogenic agars include Brilliance.TM. (Oxoid) and CHROMagar.TM. (BioMerieux).”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to replace the growth medium of Tanaka with the chromogenic agar plate as taught by Meighan.
The motivation for doing so would have been to assist in bacterial species identification. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Meighan to fully disclose “wherein the growth medium comprises chromogenic agar plates.”
Claim(s) 4, 13-14, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Wang (Early detection and classification of live bacteria using time-lapse coherent imaging and deep learning).
Regarding claim 4, Tanaka teaches “The system of claim 1,”
“wherein the image processing software is configured to receive the captured time-lapse images of the microorganisms and/or colonies thereof on the growth plate,” (Tanaka, Paragraphs 14, 22, and 40, “The method for discriminating microorganisms of the present invention uses a lensless imaging system using a photosensor and analyzes an image (“lensless image”) obtained by imaging with the system. Here, the lensless imaging system means an optical sensing system that does not go through an objective lens. The lensless imaging system used in the method for discriminating microorganisms of the present invention includes a photosensor that acquires a colony formation image (scattered light pattern) of microorganisms as pixel data, a culture vessel for culturing a sample containing microorganisms, and microorganisms. It is composed of a light source that illuminates the colony and a computing device for extracting quantitative parameters from the pixel data of the colony formation image and performing multivariate analysis using them. A device including an incubator that controls the culture temperature as needed can be a device for carrying out the method for discriminating microorganisms of the present invention. FIG. 1 shows a configuration example of an apparatus for carrying out the method of the present invention.”; “Features extracted from a colony forming images, for example, the area, brightness, the shape and the like, colony maximum growth rate mu max, colony visualized time t a (Colony appearance time), the relative average intensity I, the deviation of the histogram G, donuts property D, At least the maximum colony growth rate μ max and the relative average brightness I selected from a total of 10 parameters of image entropy H, image energy density Ed , image energy E, weighted center difference W, and average brightness C in the central region. Three or more kinds including, or at least two kinds or more including deviation G and donut property D of the histogram, or donut property D and image entropy H are calculated.”; “A CMOS sensor (DFK61BUC02, Imaging Source, Germany) was used as the image sensor. This sensor is a two-dimensional photosensor in which 3.2 μm square detection elements are arranged in 2048 × 1536, and the size of the imaging area is 6.55 mm × 4.92 mm. The company's control software (IC Capture 2.3) was used for imaging with the CMOS sensor. In addition, MATLAB (MathWorks inc., USA) was used as image analysis software. Image J was used to measure the colony area, line profile, and brightness value, and statistical analysis software R was used for cluster analysis. As a light source for illumination, a Light Emitting Diode (LED) (LM1-TPP1-01, COTCO, China) was used.”)In addition, LB medium (Merck Millipore, Germany) and low melting point agarose (Invitrogen, USA) were used for preparing the agar medium.”)
While Tanaka teaches detection of candidate microorganisms and output of species classification, (Tanaka, Paragraphs 36-37, “Microorganisms are discriminated by collating and comparing the values of the parameters calculated from colony forming images of microorganisms of unknown bacterial species with a database of the parameters for each known microorganism constructed in advance. The database is a database obtained by imaging colonization images of many bacterial species by the method described above, calculating the parameters based on the obtained image library, and creating a database. Here, when constructing a database, it is preferable to collect data from a plurality of strains even in the same bacterial species. As a method of obtaining discrimination information from the constructed database, there is a method of multivariate analysis using each parameter of the database as an explanatory variable. The multivariate analysis used in the present invention is not particularly limited, but discriminant analysis, secondary discriminant analysis, logistic regression analysis, support vector machine, decision tree, random forest, neural network, cluster analysis and the like can be adopted. As a specific example, colonies maximum growth rate mu max in the database constructed colony visualized time t a, the relative average intensity I, the deviation of the histogram G, donuts of D, the image entropy H, and the image energy density E d, Of the 10 parameters consisting of image energy E, weighted center difference W, and average brightness C in the central region, at least 3 or more including colony maximum growth rate μ max and relative average brightness I, or at least the histogram deviation G and The bacterial species can be discriminated by performing discriminant analysis using two or more kinds of parameters including donut-like D or donut-like D and image entropy H as explanatory variables to obtain a discriminant function.”), Tanaka does not expressly disclose “the image processing software configured to: (1) detect candidate microorganisms and/or colonies thereof in the time-lapse images using a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects, and (2) output a species class associated with the detected true microorganisms and/or colonies thereof using a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof.”
Wang discloses “the image processing software configured to: (1) detect candidate microorganisms and/or colonies thereof in the time-lapse images using a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects, and (2) output a species class associated with the detected true microorganisms and/or colonies thereof using a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof.” (Wang, Section “Design and training of neural networks for bacterial growth detection and classification”, Paragraphs 1 and 3-4, “We designed a two-step framework for bacterial growth detection and classification. The first step selects colony candidates with differential image analysis and refines the results with a detection DNN. We designed a pseudo-3D (P3D) DenseNet28 architecture to process our complex-valued (i.e., phase and amplitude) time lapse image stacks (see the “Methods” section). In each time-lapse imaging experiment, we used 4 time consecutive frames (4 ×0.5 = 2h) as a running window for the differential image analysis to extract individual regions of interest (ROIs) containing objects that changed their amplitude and/or phase signatures as a function of time. These initially detected objects that were extracted by the differential analysis algorithm were either growing colonies or surface impurities, e.g., from spreading the sample on the agar surface, evaporation of air bubbles in the agar plate, or coherent light speckles. We then used a DNN-based detection model to eliminate the nonbacterial objects and only kept the growing colonies (i.e., the true positives), as illustrated in Fig. 2b. We used sensitivity (or true positive rate, TPR) and precision (or positive predictive value, PPV) measurements to quantify our results.”; “The second step further classifies the species of the detected colonies with a classification DNN model following a similar network architecture. To accommodate the different growth rates of bacterial colonies, we used a longer time window in this classification neural network, containing 8 consecutive frames (8×0.5= 4h)for each sub-ROI. Since the bacterial growth detection network uses a shorter running time window of 2h, there is a natural 2-h time delay between the successful detection of a growing colony and the classification of its species. The network was trained with 7919 growing colonies, which contained 3362 E. coli, 1880K. aero genes, and 2677 K. pneumoniae colonies, and it was validated with 340 E. coli, 205 K. aerogenes, and988K. pneumoniae colonies from 6 independent plates and reached a validation classification accuracy of ~89% for E. coli, ~95%forK. aerogenes, and~98%forK. pneumoniae when the network model converged (Supplementary Fig. S2). After these network models were finalized through the training and validation data, we tested their generalization capabilities with an additional set of experiments that were never seen by the networks before; the results of these blind tests are detailed next.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to replace the multivariate image analysis software of Tanaka with the above described software of Wang including a first DNN for detection true microorganisms and/or colonies versus non-microorganism objects, and a second DNN for outputting a species class of the detected true microorganisms and/or colonies.
The motivation for doing so would have been to increase scalability to larger datasets and more micro-organism types, and additionally to overcome the need for manual parameter selection out of a limited set of parameters of multivariate analysis by using the automated feature extraction of DNN-based strategies. The multivariate analysis of Tanaka also does not account for non-microorganism objects which may cause errors in detection. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Wang to fully disclose, “the image processing software configured to: (1) detect candidate microorganisms and/or colonies thereof in the time-lapse images using a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects, and (2) output a species class associated with the detected true microorganisms and/or colonies thereof using a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof.”
Regarding claim 13, Tanaka teaches “The system of claim 12,”
“further comprising processing the time-lapsed images of the microorganisms and/or colonies thereof on the growth plate with image processing software,” (Tanaka, Paragraphs 14, 22, and 40, “The method for discriminating microorganisms of the present invention uses a lensless imaging system using a photosensor and analyzes an image (“lensless image”) obtained by imaging with the system. Here, the lensless imaging system means an optical sensing system that does not go through an objective lens. The lensless imaging system used in the method for discriminating microorganisms of the present invention includes a photosensor that acquires a colony formation image (scattered light pattern) of microorganisms as pixel data, a culture vessel for culturing a sample containing microorganisms, and microorganisms. It is composed of a light source that illuminates the colony and a computing device for extracting quantitative parameters from the pixel data of the colony formation image and performing multivariate analysis using them. A device including an incubator that controls the culture temperature as needed can be a device for carrying out the method for discriminating microorganisms of the present invention. FIG. 1 shows a configuration example of an apparatus for carrying out the method of the present invention.”; “Features extracted from a colony forming images, for example, the area, brightness, the shape and the like, colony maximum growth rate mu max, colony visualized time t a (Colony appearance time), the relative average intensity I, the deviation of the histogram G, donuts property D, At least the maximum colony growth rate μ max and the relative average brightness I selected from a total of 10 parameters of image entropy H, image energy density Ed , image energy E, weighted center difference W, and average brightness C in the central region. Three or more kinds including, or at least two kinds or more including deviation G and donut property D of the histogram, or donut property D and image entropy H are calculated.”; “A CMOS sensor (DFK61BUC02, Imaging Source, Germany) was used as the image sensor. This sensor is a two-dimensional photosensor in which 3.2 μm square detection elements are arranged in 2048 × 1536, and the size of the imaging area is 6.55 mm × 4.92 mm. The company's control software (IC Capture 2.3) was used for imaging with the CMOS sensor. In addition, MATLAB (MathWorks inc., USA) was used as image analysis software. Image J was used to measure the colony area, line profile, and brightness value, and statistical analysis software R was used for cluster analysis. As a light source for illumination, a Light Emitting Diode (LED) (LM1-TPP1-01, COTCO, China) was used.”)
While Tanaka teaches detection of candidate microorganisms and output of species classification, (Tanaka, Paragraphs 36-37, “Microorganisms are discriminated by collating and comparing the values of the parameters calculated from colony forming images of microorganisms of unknown bacterial species with a database of the parameters for each known microorganism constructed in advance. The database is a database obtained by imaging colonization images of many bacterial species by the method described above, calculating the parameters based on the obtained image library, and creating a database. Here, when constructing a database, it is preferable to collect data from a plurality of strains even in the same bacterial species. As a method of obtaining discrimination information from the constructed database, there is a method of multivariate analysis using each parameter of the database as an explanatory variable. The multivariate analysis used in the present invention is not particularly limited, but discriminant analysis, secondary discriminant analysis, logistic regression analysis, support vector machine, decision tree, random forest, neural network, cluster analysis and the like can be adopted. As a specific example, colonies maximum growth rate mu max in the database constructed colony visualized time t a, the relative average intensity I, the deviation of the histogram G, donuts of D, the image entropy H, and the image energy density E d, Of the 10 parameters consisting of image energy E, weighted center difference W, and average brightness C in the central region, at least 3 or more including colony maximum growth rate μ max and relative average brightness I, or at least the histogram deviation G and The bacterial species can be discriminated by performing discriminant analysis using two or more kinds of parameters including donut-like D or donut-like D and image entropy H as explanatory variables to obtain a discriminant function.”), Tanaka does not expressly disclose “the image processing software further configured to detect candidate microorganisms and/or colonies thereof in the time-lapse images based on differential image analysis in the time-lapse holographic images and further including a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects and a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof and outputs a species class associated with the detected true microorganisms and/or colonies thereof.”
Wang teaches “the image processing software further configured to detect candidate microorganisms and/or colonies thereof in the time-lapse images based on differential image analysis in the time-lapse holographic images and further including a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects and a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof and outputs a species class associated with the detected true microorganisms and/or colonies thereof.” (Wang, Section “Design and training of neural networks for bacterial growth detection and classification”, Paragraphs 1 and 3-4, “We designed a two-step framework for bacterial growth detection and classification. The first step selects colony candidates with differential image analysis and refines the results with a detection DNN. We designed a pseudo-3D (P3D) DenseNet28 architecture to process our complex-valued (i.e., phase and amplitude) time lapse image stacks (see the “Methods” section). In each time-lapse imaging experiment, we used 4 time consecutive frames (4 ×0.5 = 2h) as a running window for the differential image analysis to extract individual regions of interest (ROIs) containing objects that changed their amplitude and/or phase signatures as a function of time. These initially detected objects that were extracted by the differential analysis algorithm were either growing colonies or surface impurities, e.g., from spreading the sample on the agar surface, evaporation of air bubbles in the agar plate, or coherent light speckles. We then used a DNN-based detection model to eliminate the nonbacterial objects and only kept the growing colonies (i.e., the true positives), as illustrated in Fig. 2b. We used sensitivity (or true positive rate, TPR) and precision (or positive predictive value, PPV) measurements to quantify our results.”; “The second step further classifies the species of the detected colonies with a classification DNN model following a similar network architecture. To accommodate the different growth rates of bacterial colonies, we used a longer time window in this classification neural network, containing 8 consecutive frames (8×0.5= 4h)for each sub-ROI. Since the bacterial growth detection network uses a shorter running time window of 2h, there is a natural 2-h time delay between the successful detection of a growing colony and the classification of its species. The network was trained with 7919 growing colonies, which contained 3362 E. coli, 1880K. aero genes, and 2677 K. pneumoniae colonies, and it was validated with 340 E. coli, 205 K. aerogenes, and988K. pneumoniae colonies from 6 independent plates and reached a validation classification accuracy of ~89% for E. coli, ~95%forK. aerogenes, and~98%forK. pneumoniae when the network model converged (Supplementary Fig. S2). After these network models were finalized through the training and validation data, we tested their generalization capabilities with an additional set of experiments that were never seen by the networks before; the results of these blind tests are detailed next.” Note that the above steps are performed on time-lapsed holographic images (see last Paragraph of “Results”.)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to replace the multivariate image analysis software of Tanaka with the above described software of Wang including using differential image analysis-based candidate detection, a first DNN for detection true microorganisms and/or colonies versus non-microorganism objects, and a second DNN for outputting a species class of the detected true microorganisms and/or colonies.
The motivation for doing so would have been to increase scalability to larger datasets and more micro-organism types, and additionally to overcome the need for manual parameter selection out of a limited set of parameters of multivariate analysis by using the automated feature extraction of DNN-based strategies. The multivariate analysis of Tanaka also does not account for non-microorganism objects which may cause errors in detection. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka with the above teaching of Wang to fully disclose, “the image processing software further configured to detect candidate microorganisms and/or colonies thereof in the time-lapse images based on differential image analysis in the time-lapse holographic images and further including a first trained deep neural network trained to detect true microorganisms and/or colonies thereof from non-microorganism objects and a second trained deep neural network that receives as an input at least one time-lapsed image or at least one digitally processed time-lapsed image of the true microorganisms and/or colonies thereof and outputs a species class associated with the detected true microorganisms and/or colonies thereof.”
Regarding claim 14, Tanaka in view of Wang teaches “The system of claim 13,”
“wherein the microorganisms comprise a prokaryotic cell, a eukaryotic cell, bacteria, fungi, virus, multi-cellular organism, or clusters, films, or colonies thereof.” (Tanaka, Paragraphs 19-20, “In the method of the present invention, a sample containing a microorganism is cultured in a culture vessel placed on a photosensor array. Here, the "sample containing a microorganism" is a sample containing a microorganism to be discriminated and identified as a bacterial species, and may be either a clinical sample or a non-clinical sample. Clinical samples include, for example, blood, serum, plasma, blood fraction, joint fluid, urine, semen, saliva, feces, cerebrospinal fluid, gastric contents, vaginal secretions, tissue homogenate, bone marrow puncture, bone homogenate, sputum. , Suction fluid, swab and swab rinsate, other body fluids and the like. In addition, examples of non-clinical samples include substances including foods, beverages, pharmaceuticals, cosmetics, water, seawater ballasts, air, soil, sewage, plant materials, blood products, donor organs or tissue samples. In the present invention, the "microorganism" is not particularly limited as long as it forms a colony, and may be any of bacteria, fungi and the like. For example, Pseudomonas, Escherichia, Salmonella, Diarrhea, Enterobactor, Krebsiera, Seratia, Proteus, Camprovactor, Hemophilus, Morganella, Vibrio, Elsina, Asinetobacta, Stenotrophomonas. Genus, Brevendimonas, Larstonia, Achromobactor, Fuzobacterium, Prebotera, Blanchamera, Niseria, Burkholderia, Citrobacta, Hafnia, Edward Sierra, Aeromonas, Moraxera, Gram-negative bacteria such as Brucella, Pasturella, Providencia and Regionella; enterococcus, streptococcus, staphylococcus, bacillus, paenibacillus, lactic acid rod, listeria, peptstreptococcus, propionicate, crotridium Gram-positive bacteria such as Genus, Bacteroides, Gardnerella, Cochlear, Lactococcus, Leukonostock, Micrococcus, Mycobacteria and Corinebacterium; Candida, Cryptocox, Nocardia, Aokabi , Altanaria, Rhodotorula, Aspergillus, Fuzarium, Saccharomyces and Tricosporone.”)
Regarding claim 16 and 17, these claims recite a method with steps corresponding to the elements of the system recited in Claims 1, 4, and 5. Therefore, the recited steps of this claim are mapped to the analogous elements in the corresponding system claims. Additionally, the rationale and motivation to combine the Tanaka and Wang references apply here.
Regarding claim 18, Tanaka in view of Wang teaches “The system of claim 16,”
“wherein the time-lapsed images are obtained several times each hour over several hours.” (Tanaka, Paragraph 44, “Example 2 Discrimination of specific 5 bacterial species colonies (1) Construction of lensless image library Immediately after seeding each of the specific 5 bacterial species (E. coli, S. aureus, P. aeruginosa, S. enterica, C. albicans) The culture chamber was placed on a CMOS sensor, and time-lapse imaging was performed continuously for about 18 hours at intervals of 5 minutes under 37 ° C conditions. The imaging conditions were exposure time: 1/18 second, frame rate: 3.75, white balance Red: 255, Green: 154, and Blue: 64. Further, in order to reduce noise, four images were continuously acquired and the average image was used. The time when the image acquisition was started was set to t = 0h. As a result, E.I. colli, S.M. aureus, P. a. aeruginosa, S. a. For enterica, scattered light patterns thought to be derived from each bacterial colony were observed about 1-5 hours after the position where the image was not observed immediately after plating (Fig. 3A-D). After that, it was confirmed that the colony-derived scattered light pattern on the lensless image spreads horizontally with the lapse of the culture time. In addition, C.I. The pattern of albicans was observed immediately after plating into the culture chamber, and then horizontal image enlargement was observed (Fig. 3E). This is E.I. While the size of bacteria such as colli is about several μm, the eukaryotic C.I. The cell size of albicans was as large as about 10 μm in diameter, and it was considered that this was due to the fact that the cells were seeded in a state where the cell size was increased by budding. Of the colony images obtained by using two CMOS sensors, a lensless image for a total of 15 colonies randomly selected was used as a library.”)
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanaka in view of Wang further in view of Molaei.
Regarding claim 19, Tanaka in view of Wang teach “The method of claim 16,”
While Tanaka in view of Wang teaches a TFT-based sensor (see claim 1 rejection) used for hologram imaging of the microorganisms (one skilled in the art would understand the scattered light pattern generated by the imaging set-up of Figure 1 as hologram imaging; see Paragraphs 14 and 7), Tanaka does not expressly disclose the use of a lens to magnify or de-magnify these holograms.
Molaei teaches using a magnification lens magnifying holograms of microorganisms onto photosensors (Molaei, Figure 1, and Section 3.2, “The DHM includes a CW (continuous wave) He-Ne laser, collimating optics, an inverted transmission microscope, and recording CCD camera ( Fig. 1 ). We illuminate the microchannel with a collimated laser beam generated by a 7mW He-Ne laser (Lamda equals 632.8 n m). The initial beam is filtered and collimated into a beam with the diameter of 5mm by a 20X objective (Edmund Scientific), a 25-um pinhole (Thorlabs), and a 25mm diameter doublet as the collimating lens f=50mm(, Newport Inc). A 1/20-λ aluminum mirror guides the horizontal beam downward into an inverted Nikon microscope (Nikon TS-100). To record bacteria holograms, an objective at the magnification of 40X (Nikon Super Plan Fluor ELWD, NA = 0.60) is used. The objective is focused on the plane 5 away from the bottom of the microfluidics. The holograms are recorded by a 2K × 2K CCD camera (Imperx 4M15L) with a pixel resolution of 0.185/pixel, which renders the lateral resolution of 0.2 for simplicity, and were streamed continuously at the rate of 15 fps to a data acquisition computer. The exposure time was 60 . To achieve robust estimations for each hologram in a series, the typical recording lasts about 20 minutes, totaling 18,000 holograms/acquisition.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate a lens for magnifying holograms onto a photosensor, as taught by Molaei, with respect to the TFT-based hologram imaging of microorganisms of Tanaka in view of Wang.
The motivation for doing so is described by Molaei (Molaei, Introduction, Paragraph 4, “Being an inherent 3-D recording technique and its ability in recording series of holograms digitally and reconstructing holograms numerically, digital holography paves the way for studying many dynamic phenomena [20]. However, the limited spatial resolution of the earlier digital holography systems [20] that are composed of lens-less recording cameras and the laser optics have proven difficult to observe micro-scale particles over substantial depth [20]. To circumvent the recording resolution limitation, Xu et al. has developed lens-less digital holography with a point illumination to visualize intra-cellular structure of a marine diatom [21]. Later, the technique has been implemented in a submersible to track marine particulates [22] with limited success. Using partial coherent illumination and multiple projections, Ozcan et al. [23–27] have developed several portable devices to screen and detect cells based on partial coherent holography. Amid numerous variations, their fundamental system consists of a large format sensor as the substrate, over which the cells are flowing, and a point source with partial coherence [28] that illuminates the shallow suspension from multiple angles. With the advantage of high resolution digital camera and strong near field scattering, they have successfully resolved cells with resolutions of ~1μm. However, the drawbacks of lens-less systems are the shallow sample depth (<20μm) with low concentration of sample cells (<105 cells/ml) and complicated post data analysis, since the magnification of the hologram in lens-less holography depends on axial distance of the object and the source of the reference beam from the hologram [20].”) Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tanaka in view of Wang with the above teaching of Molaei to fully disclose “wherein the TFT-based image sensor captures magnified or de-magnified holograms of the microorganism objects and/or microorganism colonies thereof using a lens or set of lenses.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON JOSEPH SORRIN whose telephone number is (703)756-1565. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON JOSEPH SORRIN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights