Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/8/2026 has been entered.
Response to Amendment
Claims 1-11 and 13-15 have been amended, claims 16-18 are new and claims 1-18 are currently pending in the application.
Response to Arguments
The applicant’s arguments with respect to the rejections under 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10-12, and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Kirby et al. (US 20200357516 A1; “Kirby”) in view of Kamon (US 2021/0097331 A1; “Kamon”) and Fletcher et al. (WO 2009/088930 A2; “Fletcher”).
Regarding claim 1, Kirby teaches
An image processing apparatus (Kirby, [0016] FIG. 3 shows an example 300 of hardware that can be used to implement computing device 210 and server 220 in accordance with some embodiments of the disclosed subject matter. As shown in FIG. 3, in some embodiments, computing device 210 can include a processor 302, a display 304, one or more inputs 306, one or more communication systems 308, and/or memory 310.) comprising:
an imager (Kirby, [0013] As shown in FIG. 2, a computing device 210 can receive one or more images of a microbiological sample from a microbiological sample image source 202.; [0010] In some embodiments, image data representing pre-classified Gram-stained blood culture slides can be used to train a deep-learning neural network; the received images are generated from some type of imager and the computing device of Kirby is the processor); and
at least one processor that (Kirby fig. 2; the computing device is the processor for executing the trained model), upon execution of a program stored in memory, is configured to:
by using a trained model generated by machine learning applied to at least a part of the plurality of images, perform processing of detecting the bacterium and processing of classifying the bacterium detected through the detection processing (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected; [0013] In some embodiments, multiple CNNs can be used to recognize different types of organisms. For example, a first CNN can determine whether relatively rare bacteria are likely to be present in a blood culture Gram stain slide, and depending on the classification, the sample can be analyzed by either a second CNN that is trained to recognize relatively common types of bacteria, or a third CNN that is trained to recognize relatively rare types of bacteria.),
Kirby does not disclose:
perform control to change an imaging position of the imager relative to a sample fixing apparatus,
control the imager to change a magnification of the imager,
control the imager to generate a plurality of images by imaging one or more regions of a Gram-stained specimen of a bacterium that is held buy the sample fixing apparatus,
in a case where an automatic transmission setting of the image processing apparatus is enabled, after completing the processing of classifying the bacterium, automatically transmit, to an external apparatus, an image showing a result of the detecting and classifying the bacterium, and
in a case where the automatic transmission setting is not enabled, after completing the processing of classifying the bacterium, not automatically transmit, to the external apparatus, the image showing the result.
However, in the same art of transmitting images that have been processed for classification, Kamon, [0228], teaches that images that have processed for classification may be transmitted to another device. The images may either be transmitted based on operator instruction (i.e. automatic transmission not enabled), or automatically transmitted (i.e. automatic transmission enabled). Thus, in this combination, an image showing results of the detecting and classifying the bacterium is transmitted wither automatically or not automatically.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art for Kirby’s output of the image classification results to include the option of automatic or not automatic transmission, as taught by Kamon. The motivation is to provide an improvement to Kirby to enable optional automatic transmission of classification results to yield the predicable result of allowing the operator to have the system setup for either automatic transmission, thus requiring less operator input, or not automatic transmission, thus providing the operator greater control of transmitted results.
Additionally, in the same art of imaging bacterium samples, Fletcher, [0022]-[0023], [0113], [0216], and [0224], teaches automatic or semi-automatic control of the magnification and positioning of the microscope imager to then capture a plurality of images of regions of a Gram-stained specimen.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art for Kirby’s microscope imaging to include control to change position and magnification, as taught by Fletcher. The motivation is to enable more exact capture of images of regions of interest for analyzing the bacteria (Fletcher, [0216] and [0224]).
Regarding claim 2, Kirby, Kamon, and Fletcher further teaches the image processing apparatus according to claim 1,
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to acquire information that indicates a position of specimen relative to the sample fixing apparatus (Kirby, [0029] In some embodiments, such affirmatively selected positions can be captured in addition to images captured at preselected positions, or only images captured at the affirmatively selected positions can be used. The preselected positions are slide positions, thus are relative to the fixing apparatus) and
the type of the bacterium (Kirby, [0042] At 410, process 400 can receive an indication of the classification of each image patch from the trained CNN. For example, as each image patch is evaluated, the trained CNN can provide the probability/confidence that the patch includes an example of each of various classes. In a more particular example, the trained CNN can provide the probability/confidence that the patch includes Gram-positive cocci in chains/pairs, Gram-positive cocci in clusters, Gramnegative rods, and background.).
Regarding claim 10, the image processing apparatus according to claim 1,
Kirby further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to determine the one or more regions to be imaged (Kirby, [0029] In some embodiments, any suitable number of images can be captured from a slide/sample. For example, in some embodiments, the entire area of the slide at one or more depths of field can be captured using an automated or semi-automated process. As another example, images can be captured at various positions of a slide/sample. In such an example, the images can be captured at the same pre-defined positions on every slide, with enough images captured that it is expected that any pathogens on the slide would be present in at least a portion of the images.),
based on a user instruction (Kirby, [0040] For example, a selection tool can be executed by a computing device executing at least a portion of process 400 that can allow a user to view images of the sample and select one or more positions that appear most likely to include a positive example of at least one nonbackground class.).
Regarding claim 11, Kirby, Kamon, and Fletcher renders obvious all the claim limitations as in the consideration of claim 1 above since the operations of claim 11 are the method performed by the apparatus of claim 1.
Regarding claim 12, Kirby, Kamon, and Fletcher further teaches:
a non-transitory storage medium storing a program (Kirby, [0018] In some embodiments, memory 310 can include any suitable storage device or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 302 to present content using display 304, to communicate with server 220 via communications system(s) 308, etc.) for causing a computer to execute the image processing method according to claim 11 (see claim 11 rejection above over Kirby in view of Kamon).
Regarding claim 14, Kirby, Kamon, and Fletcher further teaches the image processing apparatus according to claim 1,
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to transmit, to the external apparatus, at least one option selected by a user from among a plurality of options including a position in the image where the bacterium classified through the classification processing exists and a reliability of the result of classification through the classification processing (Kirby, [0035] and [0046]: The transmitted classified images include location and reliability information of the bacterium).
Regarding claim 15, Kirby, Kamon, and Fletcher further teaches the image processing apparatus according to claim 1,
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to transmit, to the external apparatus, the image with data added for association with patient information (Kirby, [0035] and [0046]: The transmitted classified image includes added data of location and reliability of the bacterium detection, which is data for association with patient information because it is used in providing a diagnosis for the patient).
Regarding claim 16, Kirby, Kamon, and Fletcher further teaches the image processing apparatus according to claim 1,
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to:
determine whether the one or more regions of the Gram-stained specimen are suitable for detecting and classifying the bacterium based on the plurality of images (Kirby, [0012], [0030], and [0031]: different regions are determined to contain bacteria to classify or to be just background, thus, it is determined if the regions are suitable for detecting and classifying the bacterium).
Claims 3-8 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Kirby, Kamon, and Fletcher in view of Stumpe et al. (WO 2018/231204 A1; “Stumpe”).
Regarding claim 3, Kirby, Kamon, and Fletcher teaches the image processing apparatus according to claim 1,
Kirby further teaches
the result of the detecting and classifying the bacterium on an image that is based on the plurality of images (Kirby, [0013] a confidence associated with the determination that Gram-negative cells are present, one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Kirby and Kamon does not disclose:
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to superimpose a reliability of the result of the detecting and classifying the bacterium on an image that is based on the plurality of images.
Stumpe teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to superimpose a reliability of the result of the detecting and classifying the bacterium on an image that is based on the plurality of images (Stumpe, [pg. 17, line 11] The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.)
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching of Stumpe’s wherein the generation unit further superimposes reliability of a classification result of the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 4, Kirby, Kamon, Fletcher, and Stumpe teaches the image processing apparatus according to claim 3,
Stumpe teaches
at least one processor, upon execution of the program stored in the memory, is further configured to acquire information that indicates a position of the bacterium specimen (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching of Stumpe’s wherein the acquisition unit acquires information about the position where the bacterium classified by Gram staining exists and the type of the bacterium. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 5, Kirby, Kamon, and Fletcher teaches the image processing apparatus according to claim 4,
Kirby further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to: set a threshold value of the reliability (Kirby, [0043] The trained CNN can provide all probabilities, can provide all probabilities over a particular threshold),
in a case where the reliability of the result of the detecting and classifying the bacterium is equal to or higher than the threshold value (Kirby, [0047] At 412, process 400 can determine whether the confidence in the classification of the sample is greater than a threshold.),
in a case where the reliability of the result of the detecting and classifying the bacterium is lower than the threshold value (Kirby, [0048] Otherwise, if process 400 determines that the confidence is not greater than the threshold ("NO" at 412), process 400 can move to 416.),
not superimpose the position of the bacterium specimen and the type of the bacterium on the image that is based on the plurality of images (Kirby, [0048] Alternatively (e.g., to avoid biasing the expert), process 400 can present the images without the classification by the CNN; [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected... one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Stumpe further teaches
superimpose the position of the bacterium exists specimen (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium on the image that is based on plurality of images (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s the generation unit superimposes the position where the bacterium exists and the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 6, Kirby, Kamon, Fletcher, and Stumpe teaches the image processing apparatus according to claim 5,
Kirby further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to change the threshold value based on a user instruction (Kirby, [0064] Sensitivity and specificity were modeled as ROC curves for each classification label by varying the softmax classification thresholds required for positivity.).
Regarding claim 7, Kirby, Kamon, Fletcher, Stumpe teaches the image processing apparatus according to claim 1,
Stumpe further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to set, for each type of bacterium, whether to superimpose a position where the type of bacterium exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of bacterium on the image that is based on the plurality of images (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image}).
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s further comprising a setting unit configured to set, for each type of a bacterium, whether to superimpose the position where the bacterium exists and the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of STUMPE is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 8, Kirby, Kamon, Fletcher, Stumpe teaches the image processing apparatus according to claim 7,
Kirby further teaches
based on a user instruction (Kirby, [0040] For example, a selection tool can be executed by a computing device executing at least a portion of process 400 that can allow a user to view images of the sample and select one or more positions that appear most likely to include a positive example of at least one nonbackground class.).
Stumpe further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to set each type of bacterium, for which the position where the type of bacterium exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium are to be superimposed (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image}),
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching STUMPE’s wherein the setting unit sets a type of a bacterium for which the position where the bacterium exists and the type of the bacterium are to be superimposed. The motivation to further combine the teachings of STUMPE is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (STUMPE [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (STUMPE [pg. 3 line 5]).
Regarding claim 13, Kirby, Kamon, and Fletcher teaches the image processing apparatus according to claim 1,
KIRBY further teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to acquire information indicating a position of the bacterium (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected) and
by using a trained model generated by machine learning (Kirby, [0012] During each phase of the training process, a subset of pre-classified images can be presented to the network, which can be used to set the values of various parameters, such that the CNN automatically identifies features important for classification based on, for example, optimization of output accuracy. A trained CNN can be defined by a set of weights and biases that control the flow of information through the network such that the most discriminatory features in the images are used for classification.)
Kirby, Kamon, and Fletcher does not disclose:
that the type of the bacterium is information acquired
Stumpe teaches
the type of the bacterium is information acquired (Stumpe,[pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s the type of the bacterium is information acquired. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Claim 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kirby, Kamon, and Fletcher in view of Kubota et al. (US 20100088113 A1; “Kubota”).
Regarding claim 9, Kirby, Kamon, and Fletcher teaches the image processing apparatus according to claim 1,
Kirby, Kamon, and Fletcher fails to teach
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to transmit the image showing the result of the detecting and classifying the bacterium to a server having electronic medical charts.
However, Kubota teaches
wherein the at least one processor, upon execution of the program stored in the memory, is further configured to transmit the image to a server having electronic medical charts (Kubota, [0002] the medical image system is constituted with electronic medical chart terminals and an image server which is connected to the electronic medical chart terminals via communication circuits).
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the teachings of Kirby, Kamon, and Fletcher’s an image processing apparatus with the teachings of Kubota’s further comprising a communication unit configured to transmit data on the display image generated by the generation unit to a server having electronic medical charts. The motivation to combine the teachings of Kirby in view of Kamon and Kubota is because the references teach medical image processing where Kubota’s teaching enhances the apparatus by allowing a referring doctor to view the processed image for a simple reference at the electronic medical chart terminal via the web-compliant all-purpose communication (Kubota [0007]).
Allowable Subject Matter
Claims 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW W BEE whose telephone number is (571)270-5183. The examiner can normally be reached 9:00 - 7:00 M-Th.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights