DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of Applicant’s claim of priority from the US provisional application 63/370,066 filed on August 1, 2022.
Drawings
The 15-page drawings have been considered and placed on record in the file.
Election/Restrictions
Applicant’s election, without traverse, of Invention 3 (i.e., Claims 9-11) in the Response to Election Requirements filed on December 19, 2025 is acknowledge. Therefore, the present Office Action, only claims 1, 9-11, and 19-20 are being analyzed. Claims 2-8 and 12-18 have been withdrawn from consideration as non-elected claims.
Claim Interpretation
Regarding 10, which recites “at least one of” and lists two alternative separated by or. Under MPEP 2143.03, “All words in a claim must be considered in judging the
patentability of that claim against prior art.” In re Wilson, 424 F .2d 1382, 1385, 165
USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning
of terms as understood by one having ordinary skill in the art used in a claim will dictate
whether, and to what extent, the language limits the claim scope. Language that
suggests or makes a feature or step optional but does not require that feature or step
does not limit the scope of a claim under broadest reasonable claim interpretation. In
addition, when a claim requires selection of an element from a list of alternatives, the
prior art teaches the element if one of the alternatives is taught by prior art. See, e.g.,
Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F .3d 1288, 1298, 92 USPQ2d, 1171 (Fed.
Cir. 2009).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 9-11, and 19-20 are rejected under 35 U.S.C. 101, based on abstract idea. The claims recite a method for inspecting cryogenic electron microscopic images by capturing images as a variation of magnifications. With respect to independent method claim 1:
STEP 1: Do the claims fall within one of the statutory categories?
YES. Claim 1 is directed to a method.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?
YES, the claims are directed toward a mental process (i.e., abstract idea).
The limitation “detecting […] one or more cryogenic electron microscopy grid units that satisfy a first quality condition based on the first image montage”, “detecting […] one or more apertures within the cryogenic electron microscopy grid unit that satisfy a second quality condition based on the second image montage” and “identifying […] at least one image depicting the biological structure from the one or more images based on the Fourier transformation of each of the one or more images” as drafted, recite an abstract idea, such as a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind of a person, i.e., concepts performed in the human mind (including observation, evaluation, judgement, opinion).
As such, a person could review multiple images and analyze the images determine a grid and hole the satisfy a set criteria and identify a biological structure or an object with a degree of error or lack thereof either mentally or using a pen and paper. The mere nominal recitation that the various steps are being executed by a processor (e.g., processing unit) does not take the limitations out of the mental process grouping.
The limitation “generating a Fourier transformation of each of the one or more images“ and “generating a 3D representation of the biological structure” as drafted, recite an abstract idea, such as a process that, under broadest reasonable interpretation, covers performance of the limitation using mathematical concepts. As such, Fourier transformation and 3d generation is the application of applying mathematical calculations.
Thus, the claims recite an abstract idea.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
The additional elements of “receiving a first image montage comprising first images captured at a first magnification level, the first image montage depicting a cryogenic electron microscopy grid comprising a plurality of cryogenic electron microscopy grid units”, “receiving a second image montage comprising second images of the cryogenic
electron microscopy grid unit captured at a second magnification level”, “receiving one or more images depicting at least one of ice or a biological structure suspended within the aperture captured at a third magnification level”, and “the at least one image captured at the third magnification” are recited as mere data gathering, which may not be considered as an element which integrates the above-listed identified abstract idea into a practical application per MPEP 2106.05(g).
The additional elements “first machine learning model”, “second machine learning model”, and “third machine learning model” are recited at a high level of generality and merely equate to “apply it” or otherwise merely uses a generic computer as a tool to perform an abstract which are not indicative of integration into a practical application as per MPEP 2106.05(f). See also MPEP 2106.04(a)(2)(III) with respect to Mental Processes: “Nor do the courts distinguish between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer”. See also MPEP 2106.04(a)(2)(III)(C)(3) Using a computer as tool to perform a mental process and MPEP 2106.04(a)(2)(III)(D) as well as the case law cited therein.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
NO,
The claims herein do not include additional elements that are sufficient to amount to significantly more than the judicial exception, because as discussed above with respect to integration of the abstract idea into practical application, the additional step/element/limitation amounts to no more than an abstract idea performed on a computer. The additional elements are simply appending well-understood routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception (WURC) per MPEP 2106.05(d) and 2106.07(a)(III). Therefore, claim 1 is not patent eligible.
In addition, the elements of claims 19 and 20 are analyzed in the same manner as claim 1. The additional element recited in claims 19 and 20, i.e., “memory” and “processor”, which recited at a high level of generality and merely equate to “apply it”. Therefore independent claims 1, 19 and 20 are not patent eligible, either.
Similar analysis is made for the dependent claims 9-11, under their broadest reasonable interpretation are identified as: being either directed towards mere data gathering or an abstract idea, mental process and mathematical calculation, and not reciting additional elements that integrate the judicial exception into a practical application, and not reciting additional elements that amount to significantly more than the judicial exception.
For all of the above reasons, claims 1, 9-11, and 19-20 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, claims 1, 9-11, and 19-20 are not eligible subject matter under 35 U.S.C 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 9-11, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Carragher et al. ("Leginon: An Automated System for Acquisition of Images from Vitreous Ice Specimens") in view of Chua et al. ("Better, Faster, Cheaper: Recent Advances in Cryo-Electron Microscopy"), in further view of Wang et al. ("A FOURIER-BASED APPROACH FOR ITERATIVE 3D RECONSTRUCTION FROM CRYO-EM IMAGES").
Regarding claim 1, Carragher teaches “A method identifying biological structures depicted within an image captured using a cryogenic electron microscopy (Carragher page 1 left hand column paragraph 1 "Cryo-electron microscopy is proving to be one of the most important structural approaches in cell biological investigations. The method relies on the acquisition and analysis of large numbers of electron micrographs of macromolecular structures, which are usually preserved in vitreous ice"), the method comprising:
receiving a first image montage comprising first images captured at a first magnification level (Carragher Figure 1 and page 4 left hand column paragraph 3 "The automated system then systematically scans the grid starting with the collection of a low-magnification image of each grid square (Fig.2a)"), the first image montage depicting a cryogenic electron microscopy grid comprising a plurality of cryogenic electron microscopy grid units (Carragher page 5 left hand column paragraph 1 "After the initial calibration is completed the next step is to identify individual grid squares that are intact and uncontaminated and contain holes with suitable vitreous ice. In the automated system this is achieved by acquiring and analyzing a low-magnification (660X) image of an entire grid square (Fig.2a). Typically, there are between 500 and 1000 individual squares on the grids that are available for use");
PNG
media_image1.png
695
492
media_image1.png
Greyscale
Carragher Figure 1
detecting, (Carragher page 5 left hand column paragraph 1 "Images of each square can be acquired systematically starting from a central point or the operator may define certain areas of the grid where data acquisition is preferred") that satisfy a first quality condition (Carragher page 5 left hand column paragraph 2 "The algorithm to identify holes includes steps to (i) edge sharpen the image using a Laplacian mask; (ii) correlate the image with a synthetic template of an edge-sharpened hole; (iii)determine the coordinates of each hole center by finding a local maximum in the correlation map; (iv) check if the hole coordinates fit the local lattice that defines the hole geometry; and (v) calculate the mean and variance of the image intensity within each hole to provide an estimate of ice thickness and consistency. The size of the template is based on the diameter of the hole and is set during the initial calibration procedures") based on the first image montage (Carragher Figure 1 and page 4 left hand column paragraph 3 "The low-magnification image is first examined to determine whether the grid square is intact and free of contamination. The grid square is then analyzed to identify target holes that contain ice of suitable thickness"); and
for each of the one or more cryogenic electron microscopy grid units (Carragher page 5 left hand column paragraph 1 "Each low-magnification image is analyzed to assess the overall integrity of the grid square (is the substrate torn or damaged?) and to identify target hole locations for subsequent image acquisition at higher magnifications"):
receiving a second image montage comprising second images of the cryogenic electron microscopy grid unit captured at a second magnification level (Carragher Figure 1 and page 4 left hand column paragraph 3 "For sparsely distributed specimens, an image of each target hole is then acquired at an intermediate magnification (Fig. 2b)");
detecting, (Carragher page 5 right hand column paragraph 3 "Once holes containing ice of suitable thickness have been identified, the hole must be located at the center of the field of view, and a decision must be made as to whether the hole is suitable for further analysis") based on the second image montage (Carragher Figure 1 and page 4 left hand column paragraph 3 "This image is analyzed to detect the presence of specimen and, optionally, to target the location of the best specimen within the area. Procedures for focusing and adjusting astigmatism, as well as ensuring that the specimen is not drifting, are next performed under low-dose conditions"); and
for each of the one or more apertures (Carragher page 5 right hand column paragraph 4 "For specimens that are uniformly and densely distributed across the grid every hole is likely to contain appropriate specimen and the center of the hole thus provides a suitable target location for high-magnification image acquisition"):
receiving one or more images depicting at least one of ice or a biological structure suspended within the aperture (Carragher page 6 right hand column paragraph 2 "The result of the intermediate-magnification analysis is a set of coordinates within the hole where there are likely to be filaments of reasonable length and small curvature. If the final high-magnification images are to be acquired to film, these results are used simply to decide whether to expose the sheet of film. If the final images are to be acquired with a CCD camera, the coordinates of the center of the straightest identified filament are used to locate that filament to the center of the field of view at high magnification") captured at a third magnification level (Carragher Figure 1 and page 4 left hand column paragraph 3 "a high-magnification image is acquired, either to film or to the digital camera (Fig. 2c)");
(Carragher page 7 left hand column paragraph 4 "When using the CCD camera for high-magnification acquisition, the identified target specimen is centered in the field of view using the image shift coils on the microscope") is used
However, Carragher is not relied on to teach “using a first machine learning model”, “using a second machine learning model”, “generating a Fourier transformation of each of the one or more images”, “using a third machine learning model”, and “used for generating a 3D representation of the biological structure”.
Chua teaches “using a first machine learning model (Chua page 11 paragraph 2 "Based on desirable metrics (e.g., CTF fit, ice thickness, particle distribution, 2D classes), a neural network may then select optimal grid squares and ice thicknesses for an extended data collection or move on to screen the next grid, just as human operators now do (77)")”, “using a second machine learning model (Chua page 11 paragraph 2 "Based on desirable metrics (e.g., CTF fit, ice thickness, particle distribution, 2D classes), a neural network may then select optimal grid squares and ice thicknesses for an extended data collection or move on to screen the next grid, just as human operators now do (77)")”, and “using a third machine learning model (Chua page 11 paragraph 2 "Based on desirable metrics (e.g., CTF fit, ice thickness, particle distribution, 2D classes), a neural network may then select optimal grid squares and ice thicknesses for an extended data collection or move on to screen the next grid, just as human operators now do (77)")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a method for automating cryogenic electron microscopic image analysis as taught by Carragher to include machine learning models for analysis as taught by Chua.
The suggestion/motivation for doing so would have been that using neural networks for analyzing cryo-EM images, " These approached should increase microscope throughput and reduce the need for human intervention, therefore making the data collection process faster, more efficient, and more cost effective. This approach is intimately connected to the need to fully process the data and feed this information back to the data collection strategy" as noted by the Chua disclosure in page 12 paragraph 1.
However, the combination of Carragher and Chua is not relied on to teach “generating a Fourier transformation of each of the one or more images” and “generating a 3D representation of the biological structure”.
Wang teaches “generating a Fourier transformation of each of the one or more images (Wang page 12 paragraph 6 "Accurate Fourier-based iterative reconstruction method (FIRM) to reconstruct molecular structures from cryo-EM images")” and “generating a 3D representation of the biological structure (Wang page 2 paragraph 2 "The Fourier projection-slice theorem plays a fundamental role in all 3D reconstruction algorithms independent of whether they are implemented in real space or in Fourier space [45]. The theorem states that a slice extracted from the frequency domain representation of a 3D map yields the 2D Fourier transform of a projection of the 3D map in a direction perpendicular to the slice (Figure 1. 1). It follows from the theorem that a reconstruction can be obtained by a 3D inverse Fourier transform from the Fourier domain which is filled in by the 2D Fourier slice")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a method for automating cryogenic electron microscopic image analysis using machine learning as taught by Carragher and Chua to include Fourier transformation for generating 3d representation of a protein using cryo-EM as taught by Chua.
The suggestion/motivation for doing so would have been that addressing, " A major challenge in single particle reconstruction methods using cryo-electron microscopy is to attain a resolution sufficient to interpret fine details in three-dimensional (3D) macromolecular structures. Obtaining high resolution 3D reconstructions is difficult due to unknown orientations and positions of the imaged particles, possible incomplete coverage of the viewing directions, high level of noise in the projection images, and limiting effects of the contrast transfer function of the electron microscope" as noted by the Wang disclosure in the abstract on page 1.
Therefore, it would have been obvious to combine the disclosure of Carragher and Chua with the Wang disclosure to obtain the invention as specified in claim 1 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Regarding claim 9, the combination of Carragher, Chua, and Wang teaches ”The method of claim 1, further comprising: obtaining a second plurality of training images captured at the second magnification level, wherein the second plurality of training images each depict a cryogenic electron microscopy grid unit comprising a plurality of candidate apertures (Carragher Figure 1 and page 4 left hand column paragraph 3 "For sparsely distributed specimens, an image of each target hole is then acquired at an intermediate magnification (Fig. 2b)"), and wherein each of the second plurality of training images includes metadata indicating a location and bounding box associated with each of the plurality of candidate apertures (Carragher page 6 right hand column paragraph 2 "The result of the intermediate-magnification analysis is a set of coordinates within the hole where there are likely to be filaments of reasonable length and small curvature"); and
training the second machine learning model to identify apertures (Chua page 11 paragraph 2 "Based on desirable metrics (e.g., CTF fit, ice thickness, particle distribution, 2D classes), a neural network may then select optimal grid squares and ice thicknesses for an extended data collection or move on to screen the next grid, just as human operators now do (77)") based on the second plurality of training images (Carragher page 6 left hand column paragraph 2 "Using this algorithm correlation maps are calculated between the intermediate-magnification image and a bank of 36 filament templates. The templates in the bank are constructed from a short segment of an individual filament model that is then rotated by 5° over 0° to 180°. The template model we use consists of two uniform tracks separated by the width of the filament. The magnitude and distribution of the correlations between the image and the bank of templates provide information about the presence as well as the orientation of individual filaments").”
The proposed combination as well as the motivation for combining Carragher, Chua, and Wang references presented in the rejection of claim 1, applies to claim 9. Finally the method recited in claim 9 is met by Carragher, Chua, and Wang.
Regarding claim 10, the combination of Carragher, Chua, and Wang “The method of claim 9, wherein the second machine learning model is configured to at
least one of: determine a predicted location of a center of each of the apertures within a corresponding training image from the second plurality of training images (Carragher page 6 right hand column paragraph 2 "The result of the intermediate-magnification analysis is a set of coordinates within the hole where there are likely to be filaments of reasonable length and small curvature"); or
determine a predicted boundary of each of the apertures within a corresponding training image from the second plurality of training images (Carragher page 5 right hand column paragraph 4 "For specimens that are uniformly and densely distributed across the grid every hole is likely to contain appropriate specimen and the center of the hole thus provides a suitable target location for high-magnification Image acquisition").”
The proposed combination as well as the motivation for combining Carragher, Chua, and Wang references presented in the rejection of claim 1, applies to claim 10. Finally the method recited in claim 10 is met by Carragher, Chua, and Wang.
Regarding claim 11, the combination of Carragher, Chua, and Wang teaches “The method of claim 9, wherein training the second machine learning model comprises: inputting each of the second plurality of training images (Carragher page 6 left hand column paragraph 2 "Using this algorithm correlation maps are calculated between the intermediate-magnification image and a bank of 36 filament templates. The templates in the bank are constructed from a short segment of an individual filament model that is then rotated by 5° over 0° to 180°. The template model we use consists of two uniform tracks separated by the width of the filament. The magnitude and distribution of the correlations between the image and the bank of templates provide information about the presence as well as the orientation of individual filaments") to the second machine learning model (Chua page 11 paragraph 2 "Based on desirable metrics (e.g., CTF fit, ice thickness, particle distribution, 2D classes), a neural network may then select optimal grid squares and ice thicknesses for an extended data collection or move on to screen the next grid, just as human operators now do (77)") to obtain an output of at least one of:
a predicted location of a center of each of the apertures (Carragher page 6 left hand column paragraph 2 "(iii) Correlate the image with the bank of synthetic templates containing short segments of a filtered filament in 36 orientations. (iv) For each pixel location, compute the mean and standard deviation of the 36 filter responses"), or a predicted boundary of each of the apertures (Carragher page 5 right hand column paragraph 4 "For specimens that are uniformly and densely distributed across the grid every hole is likely to contain appropriate specimen and the center of the hole thus provides a suitable target location for high-magnification Image acquisition");
generating a comparison of the at least one of the predicted location or the predicted boundary with a predetermined location of the center of each of the apertures or a predetermined boundary of each of the apertures (Carragher page 6 left hand column paragraph 2 "For each pixel, a weighted function of the correlations is computed and compared to a threshold"); and
updating a second set of parameters of the second machine learning model based on the comparison (Carragher page 6 left hand column paragraph 2 "Create a weighted maximum correlation map by finding the maximum response of the 36 filters and z-scoring this value (subtract the mean and divide by the standard deviation). This also provides a rough estimate of the filament's orientation. (vi) Threshold the weighted maximum correlation map").”
The proposed combination as well as the motivation for combining Carragher, Chua, and Wang references presented in the rejection of claim 1, applies to claim 11. Finally the method recited in claim 11 is met by Carragher, Chua, and Wang.
Claim 19 recites a system with elements corresponding to the method with steps
recited in claim 1. Therefore, the recited elements of this claim are mapped to the
proposed combination in the same manner as the corresponding steps of method
claim 1. Additionally, the rationale and motivation to combine the Carragher, Chua, and Wang references, presented in rejection of claim 1 apply to this claim. Finally, the combination of Carragher, Chua, and Wang teaches “memory storing computer-program instructions; and one or more processors configured to execute the computer-program instructions to cause the one or more processors (Carragher page 2 left hand column paragraph 3 “Both microscopes are equipped with a Gatan cryo-transfer system. The CM200 is controlled by a UNIX workstation running the emScope software library (Kisseberth et al., 1997) and connected via a serial port to the microscope. The Gatan camera is controlled through a plug-in to the Digital Micrograph program running on a Macintosh workstation”)”.
Claim 20 recites a computer readable medium including computer executable instructions corresponding to the steps of the method recited in claim 1. Therefore, the recited instructions of the computer readable medium of claim 20 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 1. Additionally, the rationale and motivation to combine Carragher, Chua, and Wang presented in rejection of claim 1, apply to this claim. Finally, the combination of Carragher, Chua, and Wang teaches “A non-transitory computer-readable medium storing computer program instructions that, when executed, effectuate operations (Carragher page 2 left hand column paragraph 3 “Both microscopes are equipped with a Gatan cryo-transfer system. The CM200 is controlled by a UNIX workstation running the emScope software library (Kisseberth et al., 1997) and connected via a serial port to the microscope. The Gatan camera is controlled through a plug-in to the Digital Micrograph program running on a Macintosh workstation”)”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASPREET KAUR whose telephone number is (571)272-5534. The examiner can normally be reached Monday - Friday 7:30 am - 4:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASPREET KAUR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662