DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
35 U.S.C. 101 requires that a claimed invention must fall within one of the four eligible categories of invention (i.e. process, machine, manufacture, or composition of matter) and must not be directed to subject matter encompassing a judicially recognized exception as interpreted by the courts. MPEP 2106. Three categories of subject matter are found to be judicially recognized exceptions to 35 U.S.C. § 101 (i.e. patent ineligible) (1) laws of nature, (2) physical phenomena, and (3) abstract ideas. MPEP 2106(II). To be patent-eligible, a claim directed to a judicial exception must as whole be directed to significantly more than the exception itself. See 2014 Interim Guidance on Patent Subject Matter Eligibility, 79 Fed. Reg. 74618, 74624 (Dec. 16, 2014). Hence, the claim must describe a process or product that applies the exception in a meaningful way, such that it is more than a drafting effort designed to monopolize the exception. Id
Claims 1-4, 7-14, 19-23, 27 and 28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more. Claim 1 is directed to receiving a visible-light image and the fluorescence medical image of the subject; identifying a background area in at least one of the visible-light image and the fluorescence medical image trained machine-learning models; and enhancing the fluorescence medical image by suppressing a plurality of pixels in the fluorescence medical image that correspond to the identified background area, without additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, receiving a visible-light image and the fluorescence medical image of the subject, is referring to gathering data under insignificant Extra-solution activity e.g. pre-solution activity (MPEP 2106.05(g)) and identifying a background area in at least one of the visible-light image and the fluorescence medical image trained machine-learning models, referring to identifying by mental process of abstract idea, by visually observing the gathered data, the concepts are performed in the human mind including an observation, evaluation, judgment, opinion (see MPEP § 2106.04(a)(2), and by, subsection III). Also, consistent with the Applicant’s published specification, paragraph 98, the machine-learning models can be trained using a plurality of training images. The training images may include one or more annotations related to a background object/area or an object/area of interest. A person upon observation of the training images having annotations, can learn about the object of interest, and therefore can mentally observe them again in the later captured images. Claims can recite a mental process even if they are claimed as being performed on a computer i.e. the machine learning model. The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The claim further includes enhancing the fluorescence medical image by suppressing a plurality of pixels in the fluorescence medical image that correspond to the identified background area, referring to a mathematical concept of abstract idea, as consistent with Applicant’s published specification, Para [0077], the system can do so by, for example, reducing the amplitude of those pixels corresponding to the identified background area. The reduction can be done by multiplying the amplitude of each pixel with 0 or a value lower than 1. Therefore, claim 1, meets the requirement of the step 2A, prong one of the guidelines for including an abstract idea.
The claim is then considered under step 2A, prong two, for integrating the judicial exception into a practical application. Limitations that the courts have found indicative that an additional element (or combination of elements) may have integrated the exception into a practical application include:
• An improvement in the functioning of a computer, or an improvement to other technology or technical field, as discussed in MPEP §§ 2106.04(d)(1) and 2106.05(a);
• Applying or using a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, as discussed in MPEP § 2106.04(d)(2);
• Implementing a judicial exception with, or using a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim, as discussed in MPEP § 2106.05(b);
• Effecting a transformation or reduction of a particular article to a different state or thing, as discussed in MPEP § 2106.05(c); and
• Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception, as discussed in MPEP § 2106.05(e).
The courts have also identified limitations that did not integrate a judicial exception into a practical application:
• Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f);
• Adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g); and
• Generally linking the use of a judicial exception to a particular technological environment or field of use, as discussed in MPEP § 2106.05(h).
Based on the above conditions, Examiner does not believe that the language of claim 1 includes any of the qualifying conditions above. In fact, the limitations of claim 1 tend to lean more toward conditions that are not qualified i.e. “Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f)”. Therefore claim 1 fails step 2A, prong two, for not integrating the judicial exception to a practical application.
Additionally, claim 1 is considered under step 2B to include additional elements that amount to significantly more than the judicial exception. Limitations that the courts have found to qualify as "significantly more" when recited in a claim with a judicial exception include:
i. Improvements to the functioning of a computer, e.g., a modification of conventional Internet hyperlink protocol to dynamically produce a dual-source hybrid webpage, as discussed in DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258-59, 113 USPQ2d 1097, 1106-07 (Fed. Cir. 2014) (see MPEP § 2106.05(a));
ii. Improvements to any other technology or technical field, e.g., a modification of conventional rubber-molding processes to utilize a thermocouple inside the mold to constantly monitor the temperature and thus reduce under- and over-curing problems common in the art, as discussed in Diamond v. Diehr, 450 U.S. 175, 191-92, 209 USPQ 1, 10 (1981) (see MPEP § 2106.05(a));
iii. Applying the judicial exception with, or by use of, a particular machine, e.g., a Fourdrinier machine (which is understood in the art to have a specific structure comprising a headbox, a paper-making wire, and a series of rolls) that is arranged in a particular way to optimize the speed of the machine while maintaining quality of the formed paper web, as discussed in Eibel Process Co. v. Minn. & Ont. Paper Co., 261 U.S. 45, 64-65 (1923) (see MPEP § 2106.05(b));
iv. Effecting a transformation or reduction of a particular article to a different state or thing, e.g., a process that transforms raw, uncured synthetic rubber into precision-molded synthetic rubber products, as discussed in Diehr, 450 U.S. at 184, 209 USPQ at 21 (see MPEP § 2106.05(c));
v. Adding a specific limitation other than what is well-understood, routine, conventional activity in the field, or adding unconventional steps that confine the claim to a particular useful application, e.g., a non-conventional and non-generic arrangement of various computer components for filtering Internet content, as discussed in BASCOM Global Internet v. AT&T Mobility LLC, 827 F.3d 1341, 1350-51, 119 USPQ2d 1236, 1243 (Fed. Cir. 2016) (see MPEP § 2106.05(d)); or
vi. Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment, e.g., an immunization step that integrates an abstract idea of data comparison into a specific process of immunizing that lowers the risk that immunized patients will later develop chronic immune-mediated diseases, as discussed in Classen Immunotherapies Inc. v. Biogen IDEC, 659 F.3d 1057, 1066-68, 100 USPQ2d 1492, 1499-1502 (Fed. Cir. 2011) (see MPEP § 2106.05(e)).
Limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception include:
i. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP § 2106.05(f));
ii. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d));
iii. Adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea such as a step of obtaining information about credit card transactions so that the information can be analyzed by an abstract mental process, as discussed in CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011) (see MPEP § 2106.05(g)); or
iv. Generally linking the use of the judicial exception to a particular technological environment or field of use, e.g., a claim describing how the abstract idea of hedging could be used in the commodities and energy markets, as discussed in Bilski v. Kappos, 561 U.S. 593, 595, 95 USPQ2d 1001, 1010 (2010) or a claim limiting the use of a mathematical formula to the petrochemical and oil-refining fields, as discussed in Parker v. Flook, 437 U.S. 584, 588-90, 198 USPQ 193, 197-98 (1978) (MPEP § 2106.05(h)).
Based on the above conditions, Examiner is unable to identify one or more claimed elements that amount to significantly more than the judicial exception. The claim language leaning over to non-qualifying condition i.e. “ii. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry”. Therefore claim 1 fails step 2B as well for not having additional element that amounts to significantly more than the judicial exception, and hence not eligible under 101.
Regarding claim 2, reciting, displaying the enhanced fluorescence medical image, is referring to mental process of displaying abstract idea (MPEP 2106.04(a)(2)(III)(A) • a claim to "collecting information, analyzing it, and displaying certain results of the collection and analysis," where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016). Therefore, the claim is not eligible under 101.
Regarding claim 3, reciting, wherein displaying the enhanced fluorescence medical image comprises: displaying the enhanced fluorescence medical image as a part of an intraoperative video stream, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 4, reciting, displaying the visible-light image; and displaying the enhanced fluorescence medical image as an overlay on the displayed visible-light image, is referring to mental process of observing, analyzing and judgment under abstract idea. Therefore, the claim is not eligible under 101.
Regarding claim 7, reciting, receiving a first user input indicative of a degree of enhancement; and suppressing the plurality of pixels in the fluorescence medical image based on the first user input, is further referring to mathematical concept of abstract idea, mathematical concept of abstract idea of enhancing data. MPEP, 2106.04(a)(2)(I) recites, “Mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas); Digitech Image Techs., LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (holding that claims to a ‘‘process of organizing information through mathematical correlations’’ are directed to an abstract idea)”. Therefore, the claim is not eligible under 101.
Regarding claim 8, reciting, wherein identifying the background area comprises: identifying one or more background objects in the visible-light image by providing the visible-light image to the one or more trained machine-learning models, as referring to mental process of observing displayed images and correlation of an image to other trained images. Therefore, the claim is not eligible under 101.
Regarding claim 9, reciting, wherein identifying the background area comprises: identifying one or more background objects in the fluorescence image by providing the fluorescence image to the one or more trained machine learning models, as referring to mental process of observing displayed images and correlation of an image to other trained images. Therefore, the claim is not eligible under 101.
Regarding claim 10, reciting, wherein the one or more background objects comprise: an organ, a blood vessel, a connective tissue, a non-tumorous tissue, or fat, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 11, reciting, further comprising: receiving a second user input indicative of a selection of a background object of the one or more background objects; and suppressing the plurality of pixels in the fluorescence medical image based on the second user input, referring to mathematical concept of abstract idea, see claim 7 above.
Regarding claim 12, reciting, wherein identifying the background area comprises: identifying one or more objects of interest in the visible-light image by providing the visible-light image to the one or more trained machine-learning models, referring to mental process of observing an image, analyzing and judging based on model displayed training images. Therefore, the claim is not eligible under 101.
Regarding claim 13, reciting, wherein identifying the background area comprises: identifying one or more objects of interest in the fluorescence image by providing the fluorescence image to the one or more trained machine learning models, referring to mental process of observing an image, analyzing and judging based on model displayed training images. Therefore, the claim is not eligible under 101.
Regarding claim 14, reciting, wherein the one or more objects of interest comprise: a biliary structure, a tumor, an anastomotic end, a parathyroid gland, a lymph node, a limb drainage node, a thoracic duct, a renal vein, an artery, a ureter, a dysplastic tissue, a metaplastic tissue, or any combination thereof, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 19, reciting, wherein the visible-light image and the fluorescence image have been captured during a surgery, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 20, reciting, wherein the surgery comprises: a laparoscopic cholecystectomy surgery, a colorectal resection surgery, . . ., as referring to gathering data under insignificant extra-solution activity by ordinary, routine activity of tools. Therefore, the claim is not eligible under 101.
Regarding claim 21, reciting, wherein the visible-light image and the fluorescence image are sampled from one or more intraoperative videos, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 22, reciting, wherein the one or more trained machine-learning models are selected based on the surgery, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 23, reciting, wherein the visible-light image and the fluorescence image have been captured using the same camera with different filters, as referring to gathering data under insignificant extra-solution activity. Therefore, the claim is not eligible under 101.
Regarding claim 27, reciting, a system for enhancing a fluorescence medical image of a subject, comprising: similar arguments as in claim 1 is applied to this claim except the claim recites a processor and memory as part of system. The claimed invention in claim 27 is simply a concept that is performed in the human mind combined with mathematical process as discussed in claim 1 above. The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The Court concluded that the algorithm could be performed purely mentally even though the claimed procedures "can be carried out in existing computers long in use, no new machinery being necessary, (MPEP 2106.04(a)(2)(III). In the instant case, Applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept. Therefore, the claim is not eligible under 101.
Regarding claim 28, reciting, a non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, please refer to discussed corresponding method claim 1 above for further discussion. Examples of product claims reciting mental processes include: e.g. • An application program interface for extracting and processing information from a diversity of types of hard copy documents – Content Extraction, 776 F.3d at 1345, 113 USPQ2d at 1356, (MPEP 2106.04(a)(2)(III)(D). Therefore, the claim is not eligible under 101.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 23 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Specifically, the phrase “the same camera” in the claim seems to be referring to a previously cited camera which does not appear in the claim. Therefore, these citations are vague and confusing because it is unclear what feature or element is further limited by this language.
For the purposes of this office action, the phrase is treated as “a camera” in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-22, 24, 27 and 28 are rejected under 35 U.S.C. 102(a)(2) as anticipated by US 12,161,486 B2 to Freudiger et al (hereinafter ‘Freudiger’) or, in the alternative, under 35 U.S.C. 103 as obvious over US 2022/0409057 A1 to Valbusa et al (hereinafter ‘Valbusa’).
Regarding claim 1, Freudiger discloses a method of enhancing a fluorescence medical image of a subject, comprising: receiving a visible-light image and the fluorescence medical image of the subject (column 12, lines 22-27 and column 19, lines 23-27, wherein FIG. 1A provides an example of a visible light image of brain tissue treated with 5-ALA during a surgical procedure to remove a glioma. FIG. 1B shows the corresponding fluorescence image. In current practice, surgeons would resect all fluorescent tissue, and wherein the imaging system may be, e.g., a multi-channel imaging microscope that simultaneously acquires a first image at the emission wavelength of the contrast agent, e.g. the fluorescence image, and a second image outside the emission wavelength of the contrast agent); identifying a background area in at least one of the visible-light image and the fluorescence medical image by providing at least one of the visible-light image and the fluorescence medical image to one or more trained machine-learning models (column 19, lines 27-35, wherein then provides the multi-channel emission signals to an image interpretation algorithm that detects cells in the first image and suppresses non-specific background signal (e.g., auto-fluorescent background or highly-fluorescent granules), inherently as identifying the background, based on the measured spectral characteristics (e.g., by ratioing or thresholding the first image using the signal data acquired in the second image, as the corresponding visible image (e.g., at the locations corresponding to one or more cells detected in the first image)); and enhancing the fluorescence medical image by suppressing a plurality of pixels in the fluorescence medical image that correspond to the identified background area in at least one of the visible-light image and the fluorescence medical image relative to the rest of the pixels in the fluorescence medical image (column 19, lines 27-35, wherein then provides the multi-channel emission signals to an image interpretation algorithm that detects cells in the first image and suppresses non-specific background signal (e.g., auto-fluorescent background or highly-fluorescent granules) based on the measured spectral characteristics (e.g., by ratioing or thresholding the first image using the signal data acquired in the second image, as the visible image, (e.g., at the locations corresponding to one or more cells detected in the first image)). In as much as Applicant disagrees with Examiner’s assessment, Valbusa discloses identifying objects within images (Para [0051], wherein basically, deep learning is a specific type of machine learning (used to perform a specific task, in this case segmenting the reflectance image semantically, without using explicit instructions but inferring how to do so automatically from examples), which is based on neural networks.) and enhancing the data (Para [0052], wherein the flow of activity again merges at block 426 from the block 422 or the block 424. At this point, the equalizer retrieves the reflectance segmentation mask just added to its repository and the corresponding (prepared) reflectance image from its repository for determining the optical properties of the reflectance image limited to its informative region. For this purpose, the equalizer takes each pixel of the reflectance image into account; if the corresponding label in the reflectance segmentation mask is asserted (meaning that the pixel belongs to the informative region) the equalizer determines a type of the biological material represented by the corresponding pixel value (for example, blood, muscle, fat and so on depending on the color of the pixel value), and then adds its optical value (for example, ranging from 0 to 1 depending on the type of biological material and on a brightness of the pixel value) to the corresponding cell of the reflectance equalization map, whereas if the corresponding label in the reflectance segmentation mask is deasserted (meaning that the pixel belongs to the non-informative region, as suppressing the background) the equalizer adds a null value to the corresponding cell of the reflectance equalization map). Freudiger and Valbusa are combinable because they both disclose medical image object detection. Therefore, before the effective filing data of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the identifying the identifying a background area in at least one of the visible-light image and the fluorescence medical image, of Valbusa’s method with Freudiger’s in order for the segmenter directly generate the reflectance segmentation mask (para [0051]).
Regarding claim 2, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising displaying the enhanced fluorescence medical image (column 11, lines 20-21, wherein FIG. 1B provides a non-limiting example of a fluorescence image of the brain tissue, inherently as displaying).
Regarding claim 3, in the combination of Freudiger and Valbusa, Freudiger discloses wherein displaying the enhanced fluorescence medical image comprises: displaying the enhanced fluorescence medical image as a part of an intraoperative video stream (column 14, lines 46-49, wherein intraoperative use: In some instances, the disclosed methods and systems may be used intraoperatively, e.g., to guide a surgical procedure, to identify locations, inherently as visually detectable, for performing a biopsy, or to determine if resection is complete.).
Regarding claim 4, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising: displaying the visible-light image; and displaying the enhanced fluorescence medical image as an overlay on the displayed visible-light image (column 17, lines 53-57, wherein the disclosed multispectral and/or multimodal imaging methods and systems allow one to overlay images, e.g., a two-photon fluorescence image and a stimulated Raman scattering image, to provide enhanced contrast and/or to generate pseudo-color images that facilitate direct human interpretation).
Regarding claim 5, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising: extracting the luminance channel of the visible-light image; colorizing the luminance channel of the visible-light image based on the enhanced fluorescence medical image; and displaying the colorized luminance channel of the visible-light image (column 19, lines 38-42, wherein it is possible to create multi-color images based on application of a pseudo-color algorithm to the multi-channel image data (e.g., by assigning a contrast agent-associated signal to red and a background signal to green) to simplify human interpretation of the image).
Regarding claim 6, in the combination of Freudiger and Valbusa, Freudiger discloses wherein displaying the colorized luminance channel of the visible-light image comprises displaying a first color for a first organ or tissue and displaying a second color for a second organ or tissue (column 13, lines 9-11, wherein In some instances, the tissue specimen may be derived from any organ or other component of the human body, and column 23, lines 53-57, wherein the imaging methods and systems of the present disclosure may comprise the use of a pseudo-color algorithm to convert images acquired using any of one or more imaging modalities to generate multicolor images that provide, e.g., enhanced contrast for tissue structure, that provide for enhanced detection of, e.g., neoplastic tissue, and/or that facilitate human interpretation of the image. In some instances, the use of a pseudo-color algorithm to convert images acquired using any of one or more imaging modalities, inherently as applying color to multimodal images, may facilitate human interpretation of the image(s) without the need for implementing additional image interpretation algorithms).
Regarding claim 7, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising: receiving a first user input indicative of a degree of enhancement; and suppressing the plurality of pixels in the fluorescence medical image based on the first user input (column 26, lines 37-42, wherein the image interpretation algorithm provides an average signal for the contrast-positive cells after subtracting out a background signal averaged over the entire image. In some instances, the image interpretation algorithm provides the number of cells for which the signal is greater than a specified signal threshold, as inherently specified by a user, that defines a contrast-positive cell.).
Regarding claim 8, in the combination of Freudiger and Valbusa, Freudiger discloses wherein identifying the background area comprises: identifying one or more background objects in the visible-light image by providing the visible-light image to the one or more trained machine-learning models (column 19, lines 23-33, wherein the imaging system may be, e.g., a multi-channel imaging microscope that simultaneously acquires a first image at the emission wavelength of the contrast agent and a second image outside the emission wavelength of the contrast agent, e.g. a visible light image, and then provides the multi-channel emission signals to an image interpretation algorithm that detects cells in the first image and suppresses non-specific background signal (e.g., auto-fluorescent background or highly-fluorescent granules) based on the measured spectral characteristics (e.g., by ratioing or thresholding the first image using the signal data acquired in the second image, inherently as the identified background in the visible light second image).
Regarding claim 9, in the combination of Freudiger and Valbusa, Freudiger discloses wherein identifying the background area comprises: identifying one or more background objects in the fluorescence image by providing the fluorescence image to the one or more trained machine learning models (column 25, lines 14-23, wherein the image interpretation algorithm may comprise an artificial intelligence or machine learning algorithm trained, e.g., to further refine the ability of the image processing and/or image interpretation algorithm to identify individual cells within an image and/or to differentiate between normal and non-normal tissue (e.g., neoplastic tissue) based on qualitative and/or quantitative measures that are derived from the signals derived from one or more contrast agents and/or signals derived from non-fluorescence imaging modalities, inherently as including distinguishing among e.g. background unwanted signal detection).
Regarding claim 10, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the one or more background objects comprise: an organ, a blood vessel, a connective tissue, a non-tumorous tissue, or fat (column 19, lines 27-33, wherein then provides the multi-channel emission signals to an image interpretation algorithm that detects cells in the first image and suppresses non-specific background signal (e.g., auto-fluorescent background or highly-fluorescent granules) based on the measured spectral characteristics (e.g., by ratioing or thresholding the first image using the signal data acquired in the second image), inherently as prespecified cell characteristic detection and disregard other cells as background including cells within any one of inherent tissues within a human being e.g. blood vessel, connective tissue, etc.).
Regarding claim 11, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising: receiving a second user input indicative of a selection of a background object of the one or more background objects; and suppressing the plurality of pixels in the fluorescence medical image based on the second user input (column 26, lines 37-42, wherein the image interpretation algorithm provides an average signal for the contrast-positive cells after subtracting out a background signal averaged over the entire image. In some instances, the image interpretation algorithm provides the number of cells for which the signal is greater than a specified signal threshold, as inherently specified by a user, that defines a contrast-positive cell).
Regarding claim 12, in the combination of Freudiger and Valbusa, Freudiger discloses wherein identifying the background area comprises: identifying one or more objects of interest in the visible-light image by providing the visible-light image to the one or more trained machine-learning models (column 25, lines 14-23, wherein the image interpretation algorithm may comprise an artificial intelligence or machine learning algorithm trained, e.g., to further refine the ability of the image processing and/or image interpretation algorithm to identify individual cells within an image and/or to differentiate between normal and non-normal tissue (e.g., neoplastic tissue) based on qualitative and/or quantitative measures that are derived from the signals derived from one or more contrast agents and/or signals derived from non-fluorescence imaging modalities).
Regarding claim 13, in the combination of Freudiger and Valbusa, Freudiger discloses wherein identifying the background area comprises: identifying one or more objects of interest in the fluorescence image by providing the fluorescence image to the one or more trained machine learning models (column 25, lines 14-23, wherein the image interpretation algorithm may comprise an artificial intelligence or machine learning algorithm trained, e.g., to further refine the ability of the image processing and/or image interpretation algorithm to identify individual cells within an image and/or to differentiate between normal and non-normal tissue (e.g., neoplastic tissue) based on qualitative and/or quantitative measures that are derived from the signals derived from one or more contrast agents, i.e. as fluorescence, and/or signals derived from non-fluorescence imaging modalities).
Regarding claim 14, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the one or more objects of interest comprise: a biliary structure, a tumor, an anastomotic end, a parathyroid gland, a lymph node, a limb drainage node, a thoracic duct, a renal vein, an artery, a ureter, a dysplastic tissue, a metaplastic tissue, or any combination thereof (column 11, lines 30-31, wherein FIG. 3A provides a non-limiting example of a two-photon fluorescence image of a brain tumor tissue sample dosed with 5-ALA as a contrast agent.).
Regarding claim 15, in the combination of Freudiger and Valbusa, Freudiger discloses the method further comprising: enhancing the fluorescence medical image by boosting a plurality of pixels in the fluorescence medical image that correspond to the one or more objects of interest in the visible-light image (column 26, lines 53-59, wherein the imaging methods and systems of the present disclosure may comprise the use of a pseudo-color algorithm to convert images acquired using any of one or more imaging modalities to generate multicolor images that provide, e.g., enhanced contrast for tissue structure, inherently as boosting the pixels, that provide for enhanced detection of, e.g., neoplastic tissue, and/or that facilitate human interpretation of the image).
Regarding claim 16, in the combination of Freudiger and Valbusa, Valbusa discloses the trained machine-learning model is configured to output a pixel-wise mask indicative of the identified background area (Para [0051], wherein the neural network comprises basic processing elements (neurons), which perform operations based on corresponding weights; the nodes are connected via unidirectional channels (synapses), which transfer data among them. The neurons are organized in layers performing different operations, always comprising an input layer and an output layer for receiving input data and for providing output data, respectively (in this case, the reflectance image with possibly the corresponding fluorescence image and the corresponding reflectance segmentation mask, as the background mask, respectively).)
Regarding claim 17, in the combination of Freudiger and Valbusa, Valbusa discloses wherein the pixel-wise mask comprises a plurality of binary values, each binary value corresponding to a pixel in at least one of the visible-light image and the fluorescence medical image and indicative of whether the corresponding pixel depicts background or not (Para [0052], wherein he flow of activity again merges at block 426 from the block 422 or the block 424. At this point, the equalizer retrieves the reflectance segmentation mask just added to its repository and the corresponding (prepared) reflectance image from its repository for determining the optical properties of the reflectance image limited to its informative region. For this purpose, the equalizer takes each pixel of the reflectance image into account; if the corresponding label in the reflectance segmentation mask is asserted (meaning that the pixel belongs to the informative region) the equalizer determines a type of the biological material, as the object, represented by the corresponding pixel value (for example, blood, muscle, fat and so on depending on the color of the pixel value, and then adds its optical value, as the integer, (for example, ranging from 0 to 1 depending on the type of biological material and on a brightness of the pixel value) to the corresponding cell of the reflectance equalization map, whereas if the corresponding label in the reflectance segmentation mask is deasserted (meaning that the pixel belongs to the non-informative region, as the background) the equalizer adds a null value to the corresponding cell of the reflectance equalization map.)).
Regarding claim 18, in the combination of Freudiger and Valbusa, Valbusa discloses wherein the pixel-wise mask comprises a plurality of integer values, each integer value corresponding to a pixel in at least one of the visible-light image and the fluorescence medical image and indicative of an object that the corresponding pixel depicts (Para [0052], wherein the equalizer determines a type of the biological material, as the object, represented by the corresponding pixel value (for example, blood, muscle, fat and so on depending on the color of the pixel value), and then adds its optical value, as the integer, (for example, ranging from 0 to 1 depending on the type of biological material and on a brightness of the pixel value) to the corresponding cell of the reflectance equalization map, whereas if the corresponding label in the reflectance segmentation mask is deasserted (meaning that the pixel belongs to the non-informative region, inherently as the visible light) the equalizer adds a null value to the corresponding cell of the reflectance equalization map.).
Regarding claim 19, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the visible-light image and the fluorescence image have been captured during a surgery (Column 13, lines 9-11, wherein the system is used during a surgical procedure to identify locations for performing a biopsy or to determine if resection is complete.).
Regarding claim 20, in the combination of Freudiger and Valbusa, Valbusa discloses surgery comprising such as laparoscopy, arthroscopy, angioplasty (Para [0083], wherein the surgical procedure may be of any type (for example, for curative purposes, for prevention purposes, for aesthetic purposes in standard surgery, minimally invasive surgery, such as laparoscopy, arthroscopy, angioplasty, and so on).
Regarding claim 21, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the visible-light image and the fluorescence image are sampled from one or more intraoperative videos (column 20, lines 47-49, wherein the discloses systems may be configured to acquire video data for each of one or more specified excitation wavelengths, emission wavelengths, and/or imaging modalities).
Regarding claim 22, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the one or more trained machine-learning models are selected based on the surgery (column 4, lines 26-32, wherein the image interpretation algorithm comprises an artificial intelligence or machine learning algorithm. In some embodiments, the image interpretation algorithm comprises a supervised machine learning algorithm, an unsupervised machine learning algorithm, a semi-supervised machine learning algorithm, or any combination thereof, inherently as select available machine learning algorithms to choose from).
Regarding claim 24, in the combination of Freudiger and Valbusa, Freudiger discloses wherein the one or more trained machine-learning models comprise a convolutional neural network (CNN), a recurrent neural network (RNN), a diffusion model, a transformer Network, or any combination thereof (column 2, lines 38-41, wherein the machine learning algorithm comprises an artificial neural network algorithm, a deep convolutional neural network algorithm, a deep recurrent neural network, a generative adversarial network).
Regarding claim 27, in the combination of Freudiger and Valbusa, Freudiger discloses a system for enhancing a fluorescence medical image of a subject, comprising: one or more processors; one or more memories; and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors (column 5, lines 7-14, wherein system for imaging a tissue specimen comprising: a) a high-resolution optical-sectioning microscope configured to acquire images of the tissue specimen; and b) a processor configured to run an image interpretation algorithm that detects individual cells in the images acquired by the high-resolution optical-sectioning microscope and outputs a quantitative measure of a signal derived from a cell-associated contrast agent.), the one or more programs including instructions for: Please refer to the corresponding method claim 1 for further teachings.
Regarding claim 28, in the combination of Freudiger and Valbusa, Freudiger discloses a non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device (column 27, lines 47-49, wherein a non-transitory storage at any time for the software that encodes the methods and algorithms disclosed herein), cause the electronic device to: Please refer to the corresponding method claim 1 for further teachings.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Freudiger and Valbus, and further in view of US 12,426,775 B2 to Ushiroda et al (hereinafter ‘Ushiroda’).
Regarding claim 23, Freudiger and Valbusa do not specifically disclose wherein the visible-light image and the fluorescence image have been captured using the same camera with different filters. Ushiroda discloses the visible-light image and the fluorescence image have been captured using the same camera with different filters (column 2, lines 2-8, wherein an image sensor including: a pixel portion including a plurality of pixels; a first filter configured to transmit the first visible light and the fluorescence; and a second filter configured to transmit the second visible light and the fluorescence, each of the first filter and the second filter being provided on a light receiving surface of each of the plurality of pixels). Freudiger, Valbusa and Ushiroda are combinable because they all disclose medical image object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the camera with different filters of Ushiroda’s method with Freudiger’s and Valbusa’s in order to simultaneously emit the second visible light and the excitation light and to generate a background image and a fluorescence image based on the plurality of pieces of image data (column 2, lines 32-36).
Claims 25-26 are rejected under 35 U.S.C. 103 as being unpatentable over Freudiger and Valbusa, and further in view of US 12,198,329 B2 to Tenney et al.
Regarding claim 25, Freudiger and Valbusa do not specifically disclose wherein the one or more machine-learning models are trained using a plurality of training images, each training image including one or more annotations related to a background object or an object of interest. Tenney discloses (column 56, lines 13-26, wherein regarding the illustrated use of three channels in up-sampling block 790 of FIG. 7, in various embodiments, a system utilizing a CNN obtains a micro-object count from an image input. The system can do this by annotating a plurality of pixels of the input image, each pixel annotation of the set representing a probability that a corresponding pixel in the image represents the corresponding micro-object characteristic. From this analysis, a micro-object count can be obtained. In various embodiments, the plurality of micro-object characteristics comprises at least three micro-object characteristics. In various embodiments, the plurality of micro-object characteristics comprises at least a micro-object center, as object of interest, a micro-object edge, and a non-micro-object (or cell center, cell edge, and non-cell, as background object)). Freudiger, Valbusa and Tenney are combinable because they all disclose medical image object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the using of a plurality of training images, each training image including one or more annotations related to a background object or an object of interest of Tenney’s method with Freudiger’s and Valbusa’s in order to obtain micro-object count (Para [00273).
Regarding claim 26, in the combination of Freudiger and Valbusa and Tenney, Tenney further discloses wherein at least a subset of the plurality of training images comprise one or more annotations related to a pattern associated with a microstructure of a tissue (column 56, lines 16-26, wherein the system can do this by annotating a plurality of pixels of the input image, each pixel annotation of the set representing a probability that a corresponding pixel in the image represents the corresponding micro-object, as microstructure, characteristic. From this analysis, a micro-object count can be obtained. In various embodiments, the plurality of micro-object characteristics comprises at least three micro-object characteristics. In various embodiments, the plurality of micro-object characteristics comprises at least a micro-object center, as object of interest, a micro-object edge, and a non-micro-object (or cell center, cell edge, and non-cell, as background object)).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHERVIN K NAKHJAVAN whose telephone number is (571)272-5731. The examiner can normally be reached Monday-Friday 9:00-05:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sue Lefkowitz can be reached at (571)272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHERVIN K NAKHJAVAN/ Primary Examiner, Art Unit 2672
Prosecution Insights