Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office action is in response to reply filed 9/30/2025. Claims 1-11 have been amended, claims 14-15 are new and claims 1-15 are currently pending in the application.
Response to Arguments
The applicant’s arguments regarding the claim interpretation under 35 USC 112(f), the rejections under 35 USC 112(a) and 112(b), and the rejection under 35 USC 101 are persuasive. The associated interpretation and rejections have been withdrawn.
The applicant’s arguments with respect to the rejections under 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 11-12, and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kirby et al. (US 20200357516 A1; “Kirby”) in view of Kamon (US 2021/0097331 A1; “Kamon”).
Regarding claim 1, Kirby teaches
An image processing apparatus (Kirby, [0016] FIG. 3 shows an example 300 of hardware that can be used to implement computing device 210 and server 220 in accordance with some embodiments of the disclosed subject matter. As shown in FIG. 3, in some embodiments, computing device 210 can include a processor 302, a display 304, one or more inputs 306, one or more communication systems 308, and/or memory 310.) comprising:
an imager configured to generate a plurality of images by imaging a plurality of regions of a Gram-stained specimen (Kirby, [0013] As shown in FIG. 2, a computing device 210 can receive one or more images of a microbiological sample from a microbiological sample image source 202.; [0010] In some embodiments, image data representing pre-classified Gram-stained blood culture slides can be used to train a deep-learning neural network; the received images are generated from some type of imager and the computing device of Kirby is the processor); and
at least one processor that (Kirby fig. 2; the computing device is the processor for executing the trained model), upon execution of a program stored in memory, is configured to function as:
a controller configured to, by using a trained model generated by machine learning applied to at least a part of the plurality of images, perform processing of detecting a bacterium and processing of classifying the bacterium detected through the detection processing (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected; [0013] In some embodiments, multiple CNNs can be used to recognize different types of organisms. For example, a first CNN can determine whether relatively rare bacteria are likely to be present in a blood culture Gram stain slide, and depending on the classification, the sample can be analyzed by either a second CNN that is trained to recognize relatively common types of bacteria, or a third CNN that is trained to recognize relatively rare types of bacteria.),
Kirby does not disclose:
wherein the controller is configured to,
in a case where an automatic transmission setting of the image processing apparatus is enabled, after completing the processing of classifying the bacterium, automatically transmit, to an external apparatus, an image showing a result of the detecting and classifying the bacterium,
in a case where the automatic transmission setting is not enabled, after completing the processing of classifying the bacterium, not automatically transmit, to the external apparatus, the image showing the result.
However, in the same art of transmitting images that have been processed for classification, Kamon, [0228], teaches that images that have processed for classification may be transmitted to another device. The images may either be transmitted based on operator instruction (i.e. automatic transmission not enabled), or automatically transmitted (i.e. automatic transmission enabled). Thus, in this combination, an image showing results of the detecting and classifying the bacterium is transmitted wither automatically or not automatically.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art for Kirby’s output of the image classification results to include the option of automatic or not automatic transmission, as taught by Kamon. The motivation is to provide an improvement to Kirby to enable optional automatic transmission of classification results to yield the predicable result of allowing the operator to have the system setup for either automatic transmission, thus requiring less operator input, or not automatic transmission, thus providing the operator greater control of transmitted results.
Regarding claim 2, Kirby and Kamon further teaches the image processing apparatus according to claim 1,
wherein the imager acquires information about the position where the bacterium classified by Gram staining exists (Kirby, [0029] In some embodiments, such affirmatively selected positions can be captured in addition to images captured at preselected positions, or only images captured at the affirmatively selected positions can be used.) and
the type of the bacterium (Kirby, [0042] At 410, process 400 can receive an indication of the classification of each image patch from the trained CNN. For example, as each image patch is evaluated, the trained CNN can provide the probability/confidence that the patch includes an example of each of various classes. In a more particular example, the trained CNN can provide the probability/confidence that the patch includes Gram-positive cocci in chains/pairs, Gram-positive cocci in clusters, Gramnegative rods, and background.),
together with the image data (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected, a confidence associated with the determination that Gram-negative cells are present, one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Regarding claim 11, Kirby and Kamon renders obvious all the claim limitations as in the consideration of claim 1 above since the operations of claim 11 are the method performed by the apparatus of claim 1.
Regarding claim 12, Kirby and Kamon further teaches:
a non-transitory storage medium storing a program (Kirby, [0018] In some embodiments, memory 310 can include any suitable storage device or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 302 to present content using display 304, to communicate with server 220 via communications system(s) 308, etc.) for causing a computer to execute the image processing method according to claim 11 (see claim 11 rejection above over Kirby in view of Kamon).
Regarding claim 14, Kirby and Kamon further teaches the image processing apparatus according to claim 1,
wherein the controller is further configured to transmit, to the external apparatus, at least one option selected by a user from among a plurality of options including a position in the image where the bacterium classified through the classification processing exists and a reliability of the result of classification through the classification processing (Kirby, [0035] and [0046]: The transmitted classified images include location and reliability information of the bacterium).
Regarding claim 15, Kirby and Kamon further teaches the image processing apparatus according to claim 1,
wherein the controller is further configured to transmit, to the external apparatus, the image with data added for association with patient information (Kirby, [0035] and [0046]: The transmitted classified image includes added data of location and reliability of the bacterium detection, which is data for association with patient information because it is used in providing a diagnosis for the patient).
Claims 3-8 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over and Kamon in view of Stumpe et al. (WO 2018/231204 A1; “Stumpe”).
Regarding claim 3, Kirby and Kamon teaches the image processing apparatus according to claim 1,
Kirby further teaches
a classification result of the type of the bacterium on the image that is based on the image data (Kirby, [0013] a confidence associated with the determination that Gram-negative cells are present, one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Kirby and Kamon does not disclose:
wherein the controller further superimposes reliability of a classification result of the type of the bacterium on the image that is based on the image data
Stumpe teaches
wherein the controller further superimposes reliability of a classification result of the type of the bacterium on the image that is based on the image data (Stumpe, [pg. 17, line 11] The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.)
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching of Stumpe’s wherein the generation unit further superimposes reliability of a classification result of the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 4, Kirby, Kamon, and Stumpe teaches the image processing apparatus according to claim 3,
Kirby teaches
the reliability of the classification result of the type of the bacterium (Kirby, [0013] an output of microbiological sample classification system 204, such as ... a confidence associated with the determination that Gram-negative cells are present),
together with the image data (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected, a confidence associated with the determination that Gram-negative cells are present, one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Stumpe teaches
wherein the imager acquires information about the position where the bacterium classified by Gram staining exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching of Stumpe’s wherein the acquisition unit acquires information about the position where the bacterium classified by Gram staining exists and the type of the bacterium. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 5, Kirby in view of Kamon teaches the image processing apparatus according to claim 4,
Kirby further teaches
Wherein the controller is further configured to set a threshold value of the reliability (Kirby, [0043] The trained CNN can provide all probabilities, can provide all probabilities over a particular threshold),
wherein, in a case where a bacterium has reliability equal to or higher than the threshold value (Kirby, [0047] At 412, process 400 can determine whether the confidence in the classification of the sample is greater than a threshold.),
wherein, in a case where a bacterium has reliability lower than the threshold value (Kirby, [0048] Otherwise, if process 400 determines that the confidence is not greater than the threshold ("NO" at 412), process 400 can move to 416.),
the controller does not superimpose the position where the bacterium exists and the type of the bacterium on the image that is based on the image data (Kirby, [0048] Alternatively (e.g., to avoid biasing the expert), process 400 can present the images without the classification by the CNN; [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected... one or more images that were ranked by the classification system as the most likely to contain Gramnegative cells, etc.).
Stumpe further teaches
the controller superimposes the position where the bacterium exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium on the image that is based on the image data (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s the generation unit superimposes the position where the bacterium exists and the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 6, KIRBY in view of Stumpe teaches the image processing apparatus according to claim 5,
Kirby further teaches
wherein the controller changes the threshold value based on a user instruction (Kirby, [0064] Sensitivity and specificity were modeled as ROC curves for each classification label by varying the softmax classification thresholds required for positivity.).
Regarding claim 7, Kirby in view of Stumpe teaches the image processing apparatus according to claim 1,
Stumpe further teaches
wherein the controller sets, for each type of a bacterium, whether to superimpose the position where the bacterium exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium on the image that is based on the image data (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image}).
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s further comprising a setting unit configured to set, for each type of a bacterium, whether to superimpose the position where the bacterium exists and the type of the bacterium on the image that is based on the image data. The motivation to further combine the teachings of STUMPE is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Regarding claim 8, Kirby in view of Stumpe teaches the image processing apparatus according to claim 7,
Kirby further teaches
based on a user instruction (Kirby, [0040] For example, a selection tool can be executed by a computing device executing at least a portion of process 400 that can allow a user to view images of the sample and select one or more positions that appear most likely to include a positive example of at least one nonbackground class.).
Stumpe further teaches
wherein the controller sets a type of a bacterium for which the position where the bacterium exists (Stumpe, [pg. 17, line 11] Figure 3A is an illustration of the field of view 150 of a microscope showing a prostate cancer specimen at a given magnification level, for example 10X, as it would be in a conventional microscope without the capability of this disclosure. Figure 3B is an illustration of an augmented field of view 150 seen by the pathologist using the microscope of Figure 1, with an enhancement in the form of an outline 156 superimposed on the field of view circumscribing cells in the sample which are likely to be cancerous. The enhancement further includes a text box 158 providing annotations, in this example Gleason score grading and size measurements.) and
the type of the bacterium are to be superimposed (Stumpe, [pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image}),
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching STUMPE’s wherein the setting unit sets a type of a bacterium for which the position where the bacterium exists and the type of the bacterium are to be superimposed. The motivation to further combine the teachings of STUMPE is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of STUMPE enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (STUMPE [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (STUMPE [pg. 3 line 5]).
Regarding claim 13, Kirby and Kamon teaches the image processing apparatus according to claim 1,
KIRBY further teaches
wherein information indicating the position where the bacterium classified by Gram staining exists (Kirby, [0013] an output of microbiological sample classification system 204, such as an indication that the presence of Gram-negative cells has been detected) and
by using a trained model generated by machine learning using the image data (Kirby, [0012] During each phase of the training process, a subset of pre-classified images can be presented to the network, which can be used to set the values of various parameters, such that the CNN automatically identifies features important for classification based on, for example, optimization of output accuracy. A trained CNN can be defined by a set of weights and biases that control the flow of information through the network such that the most discriminatory features in the images are used for classification.)
Kirby and Kamon does not disclose:
that the type of the bacterium is information acquired
Stumpe teaches
the type of the bacterium is information acquired (Stumpe,[pg. 17, line 18] In this particular example, the annotations are that 87 percent of the cells within the outline are Gleason grade 3 score, 13 percent of the cells are Gleason 20 grade 4 score, and the tumor composed of cells of Gleason grade 4 score has a diameter of 0.12 μm.; see [Fig. 3B] {showing superimposed labels/classifications on the microscopy image})
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the teaching Stumpe’s the type of the bacterium is information acquired. The motivation to further combine the teachings of Stumpe is because the references teach the classification and sharing of biological samples derived from microscopy where the teaching of Stumpe enhance the apparatus by assisting clinicians in characterizing samples by focusing their attention to areas of interest (Stumpe [pg. 13 line 11]), ultimately improving diagnoses and staging for patients (Stumpe [pg. 3 line 5]).
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Kirby and Kamon in view of Kubota et al. (US 20100088113 A1; “Kubota”).
Regarding claim 9, Kirby in view of Kamon teaches the image processing apparatus according to claim 1,
Kirby in view of Kamon fails to teach
further comprising a communication unit configured to transmit data on the display image generated by the generation unit to a server having electronic medical charts.
However, Kubota teaches
further comprising a communication unit configured to transmit data on the display image generated by the generation unit to a server having electronic medical charts (Kubota, [0002] the medical image system is constituted with electronic medical chart terminals and an image server which is connected to the electronic medical chart terminals via communication circuits).
Therefore, it would have been obvious to persons of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the teachings of Kirby in view of Kamon’s an image processing apparatus with the teachings of Kubota’s further comprising a communication unit configured to transmit data on the display image generated by the generation unit to a server having electronic medical charts. The motivation to combine the teachings of Kirby in view of Kamon and Kubota is because the references teach medical image processing where Kubota’s teaching enhances the apparatus by allowing a referring doctor to view the processed image for a simple reference at the electronic medical chart terminal via the web-compliant all-purpose communication (Kubota [0007]).
Regarding claim 10, the image processing apparatus according to claim 9,
Kirby further teaches
further comprising a determination unit configured to determine a region to be imaged of the Gram-stained specimen (Kirby, [0029] In some embodiments, any suitable number of images can be captured from a slide/sample. For example, in some embodiments, the entire area of the slide at one or more depths of field can be captured using an automated or semi-automated process. As another example, images can be captured at various positions of a slide/sample. In such an example, the images can be captured at the same pre-defined positions on every slide, with enough images captured that it is expected that any pathogens on the slide would be present in at least a portion of the images.),
based on a user instruction (Kirby, [0040] For example, a selection tool can be executed by a computing device executing at least a portion of process 400 that can allow a user to view images of the sample and select one or more positions that appear most likely to include a positive example of at least one nonbackground class.),
wherein the communication unit transmits information indicating the region to be imaged to an imaging apparatus that images the Gram-stained specimen (Kirby, [0018] In some such embodiments, processor 302 can execute at least a portion of the computer program to present content ( e.g., images, user interfaces, graphics, tables, etc.), receive content from server 220, transmit information to server 220, execute at least a portion of a microbiological sample classification program, etc.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW W BEE whose telephone number is (571)270-5183. The examiner can normally be reached 9:00 - 7:00 M-Th.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677