DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-7, 9-11, 15, and 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Stumpe (U.S. Pub. No. 2021/0018742).
Re claim 1: Stumpe disclose a computer-based method for analyzing a plurality of images, in particular of diagnostic and/or medical images (i.e., “improved microscope system and method for assisting a pathologist in classifying biological samples such as blood or tissue, e.g., as containing cancer cells or containing a pathological agent such as plasmodium protozoa or tuberculosis bacteria”, Paragraph [0002]), comprising the steps:
determining a first set of requirements (See for example, “continue to provide enhancements to the field of view and assist the pathologist in characterizing or classifying the specimen in substantial real time as the operator navigates around the slide (e.g., by use of a motor 116 driving the stage), by changing magnification by switching to a different objective lens 108A or 108B, or by changing depth of focus by operating the focus knob 160”, Paragraph [0045]; and “assuming multiple different pattern recognizers are loaded into the compute unit, an automatic specimen type detector or manual selector switches between the specimen dependent pattern recognition models (e.g. prostate cancer vs breast cancer vs malaria detection)”, Paragraph [0058]);
obtaining a plurality of images of a specimen from an imaging device (i.e., “The camera may take the form of a high resolution (e.g., 16 megapixel) video camera operating at say 10 or 30 frames per second. The digital camera captures magnified images of the sample as seen through the eyepiece of the microscope. Digital images captured by the camera are supplied to a compute unit 126”, Paragraph [0043]), in particular in a microscope (i.e., “conventional pathologist microscope 102”, Paragraph [0040]), based on the first set of requirements (i.e., “capturing with a camera 124 a digital image of the sample as seen by the user through the eyepiece of the microscope, using a machine learning pattern recognizer (200, FIG. 5, FIG. 8) to identify areas of interest in the sample from the image captured by the camera 124, and superimposing an enhancement to the view of the sample as seen by the user through the eyepiece of the microscope as an overlay”, Paragraph [0047]);
obtaining a second set of requirements (See for example, “the user can specify the type of annotations or enhancements they wish to see projected onto the field of view, thereby giving the user control as to how they wish the microscope to operate in augmented reality mode. For example, the user could specify enhancements in the form of heat map only. As another example, if the specimen is a blood sample, the user could specify enhancements in the form of rectangles identifying plasmodium present in the sample. In a prostate sample, the user can specify boundaries our outlines surrounding cells which a Gleason score of 3 or more, as well as annotations such as shown and described previously in FIG. 3B”, Paragraph [0107]); and
analyzing the specimen from the plurality of images based on the second set of requirements (i.e., Paragraphs [0071]-[0076]; “the image of the field of view is provided as input to the relevant machine learning pattern recognizer 200 in the compute unit 126 (FIG. 5) to perform inference”, Paragraph [0105]; and “the graphics card or GPU 206 in the compute unit 126 generates digital image data corresponding to the enhancement or augmentation relevant to the sample type and this digital image data is provided to the AR display unit 128 for projection onto the field of view for viewing by the pathologist in the eyepiece 104”, Paragraph [0106]).
Re claim 2: Stumpe disclose wherein the first set of requirements comprises one or more requirements related to a content of one or more images (See for example, Paragraph [0081]).
Re claim 3: Stumpe disclose wherein the first set of requirements comprises one or more of the following: a spatial position; an identification of a sample-related carrier; a time-related parameter; a wavelength-related parameter; an intensity difference; a movement parameter; and a pattern (i.e., “a “pre-scan” mode in which the motor 116 drives the microscope slide to a series of X-Y positions and obtains low magnification images with the camera at each position”, Paragraph [0081]).
Re claim 4: Stumpe disclose wherein the first set of requirements comprises one or more requirements related to a form of one or more images (i.e., “changing magnification by switching to a different objective lens 108A or 108B”, Paragraph [0045]).
Re claim 5: Stumpe disclose wherein the first set of requirements comprises one or more of the following: an image size; an image aperture; and an image resolution (See for example, Paragraph [0048]).
Re claim 6: Stumpe disclose wherein the obtaining of the plurality of images comprises the step: storing a subset of images provided by an imaging system (i.e., “it may be desirable for the pathologist to make records of their work in characterizing or classifying the sample. Such records could take the form of digital images of the field of view (with or without enhancements) which can be generated and stored (e.g., in the memory 212 of the compute unit) and then transmitting them via interface 216 to the attached pathology workstation 140, The workstation software will typically include workflow software that the pathologist follows in performing a classification or characterization task on a sample and generating a report. Such software includes a tool, e.g., icon or prompts, which permit the pathologist to insert into the report the stored digital images of the field of view and relevant annotations or enhancements which are stored in the memory 212”, Paragraph [0086]).
Re claim 7: Stumpe disclose wherein the obtaining of the plurality of images comprises the step: controlling an imaging system for capturing images (i.e., “the microscope 102 includes a capability to identify which microscope objective lens is currently in position to image the sample, e.g., with a switch or by user instruction to microscope electronics controlling the operation of the turret containing the lenses, and such identification is passed to the compute unit 126 using simple electronics so that the correct machine learning pattern recognition module in an ensemble of pattern recognizers (see FIG. 8 below) is tasked to perform inference on the new field of view image”, Paragraph [0049]).
Re claim 9: Stumpe disclose wherein the obtaining a plurality of images is based on a machine learning algorithm (i.e., “the compute unit 126 includes a machine learning pattern recognizer which receives the images from the camera”, Paragraph [0044]).
Re claim 10: Stumpe disclose wherein the machine learning algorithm is based on the first set of requirements and/or on the second set of requirements (i.e., “such identification is passed to the compute unit 126 using simple electronics so that the correct machine learning pattern recognition module in an ensemble of pattern recognizers (see FIG. 8 below) is tasked to perform inference on the new field of view image”, Paragraph [0049]).
Re claim 11: Stumpe disclose wherein the machine learning algorithm comprises a neural network, in particular a convolutional neuronal network (i.e., “the compute unit includes a deep convolutional neural network pattern recognizer 200 in the form of a memory 202 storing processing instructions and parameters for the neural network and a central processing unit 204 for performance of inference on a captured image”, Paragraph [0056]; and Paragraph [0059]).
Re claim 15: Stumpe disclose an imaging device (i.e., “augmented reality microscope system 100 for pathology”, Paragraph [0040]), in particular a microscope, configured for:
executing the method according to claim 1 (Refer to claim 1 above).
Re claim 16: Stumpe disclose a graphical user interface (i.e., “simple user interface on the compute unit 126”, Paragraph [0052]), configured to:
define content-related and/or format-related requirements for an image acquisition (See for example, “continue to provide enhancements to the field of view and assist the pathologist in characterizing or classifying the specimen in substantial real time as the operator navigates around the slide (e.g., by use of a motor 116 driving the stage), by changing magnification by switching to a different objective lens 108A or 108B, or by changing depth of focus by operating the focus knob 160”, Paragraph [0045]; and “assuming multiple different pattern recognizers are loaded into the compute unit, an automatic specimen type detector or manual selector switches between the specimen dependent pattern recognition models (e.g. prostate cancer vs breast cancer vs malaria detection)… The microscope monitors which lens is placed by the user into the optical path and communicates the selection to the compute unit”, Paragraph [0058]);
display one or more images according to the defined requirements (i.e., “The image 1522 displayed is the current field of view of the microscope”, Paragraph [0157]);
interface (See for example, Paragraph [0056]) with the imaging device according to claim 15 (Refer to claim 15 above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Stumpe in view of Jaffray et al. (U.S. Pub. No. 2020/0002662). The teachings of Stumpe have been discussed above.
As to claim 8, Stumpe does not explicitly disclose wherein the obtaining of the plurality of images comprises an event-triggered obtaining of one or more images.
Jaffray et al. teaches obtaining of the plurality of images comprises an event-triggered obtaining of one or more images (See for example, “The imaging system 102 may have a programmable microscope that is capable of generating serial images of living cells in multi-well plates before and after irradiation”, Paragraph [0090]; and “device 100 triggers the therapy system 106 to execute the therapeutic protocol for radiation therapy and/or drug therapy. At 612, device 100 collects images and processes the images for therapeutic analysis”, Paragraph [0097]).
Stumpe and Jaffray et al. are analogous art because they are from the field of digital image processing for specimen analysis.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Stumpe by incorporating the event-triggered obtaining of one or more images, as taught by Jaffray et al.
The suggestion/motivation for doing so would have been to compute and monitor a therapeutic protocol by measuring sensitivity of cell growth to treatment based on a radiation therapy protocol.
Therefore, it would have been obvious to combine Jaffray et al. with Stumpe to obtain the invention as specified in claim 8.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Stumpe in view of Cotte et al. (U.S. Pub. No. 2019/0251330). The teachings of Stumpe have been discussed above.
As to claim 12, Stumpe teaches wherein the step obtaining a plurality of images from an imaging device comprises: obtaining a plurality of images from a first device (i.e., “capturing with a camera 124 a digital image of the sample as seen by the user through the eyepiece of the microscope, using a machine learning pattern recognizer (200, FIG. 5, FIG. 8) to identify areas of interest in the sample from the image captured by the camera 124, and superimposing an enhancement to the view of the sample as seen by the user through the eyepiece of the microscope as an overlay”, Paragraph [0047]).
However, Stumpe does not explicitly disclose obtaining a plurality of images from an imaging device comprises: obtaining a plurality of images from one or more further devices.
Cotte et al. teaches obtaining a plurality of images from an imaging device comprises: obtaining a plurality of images from one or more further devices (See for example, “a microscopic object characterization system 1 comprises a computer system 2, 2a, 2b and a microscope 4 with a computing unit 5 connected to the computer system, whereby the microscopic object characterization system 1 may comprise a plurality of computer systems and a plurality of microscopes connected via a global communications network”, Paragraph [0120]; and Paragraph [0486]).
Stumpe and Cotte et al. are analogous art because they are from the field of digital image processing for specimen analysis.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Stumpe by incorporating the obtaining of a plurality of images from one or more further devices, as taught by Cotte et al.
The suggestion/motivation for doing so would have been to store and analyze microscope object characterization data using one or more data centers connected via the internet.
Therefore, it would have been obvious to combine Cotte et al. with Stumpe to obtain the invention as specified in claim 12.
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Stumpe in view of Yaguchi et al. (U.S. Pub. No. 2015/0179396). The teachings of Stumpe have been discussed above.
As to claim 13, Stumpe does not explicitly disclose parallel to the obtaining a plurality of images of the specimen, comprising the step: controlling an influence on the specimen.
Yaguchi et al. teaches parallel to the obtaining a plurality of images of the specimen, comprising the step: controlling an influence on the specimen (i.e., “A desired gas-liquid flow rate and a desired pressure in an environment in the vicinity of the specimen are inputted in the specimen-environment-control unit 10 and are set to be desired conditions in the supply device 19. Gas, liquid, and a mixed fluid thereof are supplied from the supply device 19, and the gas, the liquid, and the mixed fluid are introduced into the capillary 17. A transmission image of the specimen 23 in an environment of the gas, the liquid, and the mixed fluid is captured by the TV camera 8. The transmission image is displayed in the image display unit 9a and is continuously recorded in the image recording unit 9b. In the specimen-environment-control unit 10, an environment condition in the vicinity of the specimen 23 is constantly monitored and is recorded while being synchronized with a counter of the image recording unit 9b”, Paragraph [0025]).
Stumpe and Yaguchi et al. are analogous art because they are from the field of digital image processing for specimen analysis.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Stumpe by incorporating the parallel to the obtaining a plurality of images of the specimen, comprising the step: controlling an influence on the specimen, as taught by Yaguchi et al.
The suggestion/motivation for doing so would have been to observe a specimen in a gas or liquid environment and a reaction of the specimen at a high resolution and dynamically observe a change of the specimen.
Therefore, it would have been obvious to combine Yaguchi et al. with Stumpe to obtain the invention as specified in claim 13.
As to claim 14, Yaguchi et al. teaches wherein the influence on the specimen is a provision of a liquid and/or a mechanical pressure (i.e., “A desired gas-liquid flow rate and a desired pressure in an environment in the vicinity of the specimen are inputted in the specimen-environment-control unit 10 and are set to be desired conditions in the supply device 19. Gas, liquid, and a mixed fluid thereof are supplied from the supply device 19, and the gas, the liquid, and the mixed fluid are introduced into the capillary 17”, Paragraph [0025]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSE M TORRES whose telephone number is (571)270-1356. The examiner can normally be reached Monday thru Friday; 10:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSE M TORRES/Examiner, Art Unit 2664 07/23/2026
Prosecution Insights