DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments filed on 4/13/26 have been considered but are moot because the arguments do not apply to any of the references being used in the current rejection. The amendment necessitates the new ground(s) of rejection presented due to the added language in the independent claim(s).
Claim Rejections – 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
PNG
media_image1.png
158
934
media_image1.png
Greyscale
Claim(s) 1, 6-21 is/are rejected under 35 U.S.C. § 103 as being unpatentable over Brogden et al. (US 20170256380 A1) [hereinafter Brogden] in view of Uemoto et al. (US 20170122852 A1) [hereinafter Uemoto], Tanaka et al. (US 20080315097 A1) [hereinafter Tanaka], Amthor et al. (US 20220091405 A1) [hereinafter Amthor], and https://en.wikipedia.org/w/index.php?title=Convolutional_neural_network (archived 08/12/21) [hereinafter Wikipedia].
Regarding claim 1, Brogden teaches a method comprising:
imaging (see e.g. ion beam imaging, [0097-98]), using a charged particle microscope (see e.g. [0097-98], fig 16: 1604, fig 24: 2402, 2404, [0050,189]), a grid (see e.g. TEM grid, fig 24: 2412, [0082]) positioned on a stage of the charged particle microscope to obtain an image (see fig 24), the grid including a support portion (e.g. top part of 2412) and a plurality of posts (see in fig 24, fig 27: 610) extending from the support portion,
determining, with a processing device (required for intended operation of system), based on the image, a designated weld location for each post of the plurality of posts (see target attachment point, e.g. [0095]);
Brogden may fail to explicitly disclose determining, with the processing device, based on the image, whether the designated weld location of each post of the plurality of posts is defective.
However, Uemoto teaches a system to determine whether a post is deformed or damaged, and to designate unusable post locations to skip for future use (see Uemoto, [0084]), said system comprising determining, with the processing device, based on the image, whether the designated weld location of each post of the plurality of posts is defective (see [0084], weld locations on that post would be deemed defective). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to combine the teachings of Uemoto in the system of the combined prior art, because a skilled artisan would have been motivated to look for ways to designate unusable post locations, in the manner taught by Uemoto.
Brogden may fail to explicitly disclose storing a stage location associated with each weld location that is not defective.
However, the use of storing observation conditions for a specimen to facilitate future observation, rather than trying to find the lamella from scratch each time, was well known in the art at the time the application was effectively filed. For example, Tanaka teaches a system for storing a stage location associated with each valid sample location (see Tanaka, e.g. [0049]) which enables the ability to automatically and efficiently acquire information from multiple samples and/or different sample holders (see [0005], fig. 14). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to combine the teachings of Tanaka in the system of the prior art because a skilled artisan would have been motivated to store more information about each sample, including information about known effective imaging parameters including stage position, corresponding to multiple samples and thereby associated with each weld positions, in order to try to improve efficiency of operation with multiple samples.
The combined teaching of Brogden and Uemoto may fail to explicitly disclose wherein determining the designated weld location for each post of the plurality of posts and determining whether the designated weld location of each post of the plurality of posts is defective is performed using a plurality of connected convolutional neural networks (CNNs), wherein the plurality of connected CNNs includes a first model operable to segment the image and output a segmented image, and wherein the plurality of connected CNNs includes a second model operable to receive the segmented image from the first model and output an indication of whether one or more designated weld locations of the plurality of posts are defective.
However, Amthor teaches a system for assisting in aligning and centering imaging of regions based on sample carrier features (see Amthor, abstract, [0004-05]), said system comprising using a plurality of connected convolutional neural networks (CNNs) (see CNN and additionally using plurality of neural networks, [0015]), wherein the plurality of connected CNNs includes a first model operable to segment the image and output a segmented image (see first neural net to generate a segmentation mask, which is a CNN, [0015,92]), and wherein the plurality of connected CNNs includes a second model (see generally the e.g. another neural network for classification, [0015]) operable to receive the segmented image from the first model and output an indication of whether one or more designated [feature] locations of the plurality of posts are [identified]. It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to combine the teachings of Amthor in the system of the prior art, thereby providing image segmentation prior to the analysis of weld locations, because a skilled artisan would have been motivated to improve automating, aligning, and centering images to focus in on sample carrier features, and avoid unnecessary analysis of unrelated regions not related to the e.g. background, in the manner taught by Amthor.
It is noted that the combined teaching of Brogden, Uemoto, and Amthor may fail to explicitly disclose that the second model is for outputting that the weld locations (welded features at the designated location are defective) are defective (not just generic features). However, some kind of image recognition would have been required for the intended operation of identifying that the sample holder is deformed or damaged (see Uemoto, [0083-84]; e.g. using template matching, [0213]), and the use of convolutional neural networks for effective image recognition was well known in the art at the time the application was effectively filed. For example, Wikipedia teaches using CNNs to analyze visual imagery and for image recognition and classification (see e.g. Wikipedia, p1). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to try to use CNNs for automating the image recognition of the features taught by the prior art. It is noted that broadly providing an automatic or mechanical means to replace a manual activity which accomplishes the same result does not differentiate the claimed apparatus from a prior art apparatus. See In re Venner, 262 F.2d 91, 95, 120 USPQ 193, 194 (CCPA 1958).
Regarding claim 6, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches the first model (redefining the first model as a model comprising both the segmentation model of Amthor and the defect/damage identification system of Uemoto (implemented as its own CNN); and also redefining the second model as a routine (well-known) output handling CNN model to manage output of the indications, e.g. controlling data storage to a database (e.g. whatever storage system/format used by Tanaka)) is trained to identify defects associated with posts configured to receive lamella (see Uemoto, [0084]).
Regarding claim 7, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches the first model is a segmentation model (see Amthor, e.g. [0015]).
Regarding claim 8, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches defective includes bent, tilted or rotated (determination of damaged/unusable position will naturally occur when the defective weld location is bent, tilted, or rotated, see e.g. Uemoto, [0084]).
Regarding claim 9, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches wherein defective includes missing material (determination of damaged/unusable position will naturally occur when the defective weld location is missing material, see e.g. Uemoto, [0084]).
Regarding claim 10, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor may fail to explicitly disclose the second model is operable to determine, based on the segmented image, whether a lamella is already present at the weld location of each post. However, given the alignment and imaging of the attachment position (see Brogden, e.g. figs 24, 27), it would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to not attach a lamella on top of an existing lamella on the same spot during imaging; and similarly, automating this process would also have been obvious as a routine skill in the art. Further, Amthor teaches that analysis of the object may also be accomplished in addition to the analysis of the holding frame (see Amthor, e.g. [0025]). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to set up the system to automatically analyze an expected lamella, and output areas having an absence of the lamella sample during analysis.
Regarding claim 11, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches the second model is trained to identify whether the lamella is already present at the weld location of each post (see discussion per claim 10 above).
Regarding claim 12, the combined teaching of Brogden, Uemoto, Tanaka, Amthor, Brogden II, and Wikipedia may fail to explicitly disclose the second model is a segmentation model. However, Amthor teaches that non-semantic and semantic segmentation models can utilized to accomplish the intended operation of segmenting the image and separating background from identified elements (see Amthor, e.g. [0094]). Under the broadest reasonable interpretation of the claims, the claimed second model may be read as additionally including the second segmentation model of Amthor.
Regarding claim 13, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is valid (see Uemoto, [0084], determines validity/damage to the holder).
Regarding claim 14, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is located in a designated location (determining attachment position of TEM grid, moving both until relative location is in a designated position relative to the lamella, see Brogden, [0095]).
Regarding claim 15, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is located in the designated location (see Amthor, e.g. [0015,26]) is performed using an artificial neural network trained to identify whether the grid is located in the designated location (see e.g. [0015,26]).
Regarding claim 16, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches the artificial neural network is a convolutional neural network (see Amthor, [0015]).
Regarding claim 17, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is tilted (required for rotationally aligning sample with TEM grid, see e.g. Brogden, claim 53).
Regarding claim 18, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is rotated (required for rotationally aligning sample with TEM grid, see e.g. Brogden, claim 53).
Regarding claim 19, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches determining, based on the image, whether the grid is flipped (required for rotationally aligning sample with TEM grid, see e.g. Brogden, claim 53).
Regarding claim 20, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches automatically navigating to each weld location for welding of a lamella (see e.g. Tanaka, [0005]; Brogden, claim 26)
Regarding claim 21, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor teaches the indication designates whether the designated weld location of each post of the plurality of posts is defective (see Uemoto, [0084]; the designated weld location is set to be a detective location (see Tanaka) because the post found there is deformed or damaged).
Claim(s) 2-4 is/are rejected under 35 U.S.C. § 103 as being unpatentable over Brogden, Uemoto, Tanaka, and Amthor, as applied to claim 1 above, in further view of Brogden (US 20150348751 A1) [hereinafter Brogden II].
Regarding claim 2, the combined teaching of Brogden, Uemoto, Tanaka, and Amthor may fail to explicitly disclose the second model is operable to determine, based on the segmented image, whether there is contamination present on or around the weld location. However, Brogden II teaches it was well known in the art to identify contamination defects during observation of thinly sliced samples (see Brogden II, [0002]), and teaches a system to use machine vision on multiple images (see e.g. [0047]) to compensate for topographical variations in the lamella (see [0005]). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to combine the teachings of Brogden II in the system of the prior art, to provide detection of other problems such as contamination, after the segmented image is output (i.e. after limiting analysis to the sample holder), because a skilled artisan would have been motivated to look for ways to identify contamination type defects, while better compensating for topographical variations, in the manner taught by Brogden II.
It is noted that the teaching may fail to explicitly disclose training a CNN to detect the contamination. However, it is noted that the “plurality of connected CNNs” may read on a system including CNNs but using different subsystems for image detection. Further, it is noted that Wikipedia teaches using CNNs to analyze visual imagery and for image recognition and classification (see e.g. Wikipedia, p1). It would have been obvious to a person having ordinary skill in the art at the time the application was effectively filed to try to use CNNs for automating the image recognition of the features taught by the prior art. It is noted that broadly providing an automatic or mechanical means to replace a manual activity which accomplishes the same result does not differentiate the claimed apparatus from a prior art apparatus. See In re Venner, 262 F.2d 91, 95, 120 USPQ 193, 194 (CCPA 1958).
Regarding claim 3, the combined teaching of Brogden, Uemoto, Tanaka, Amthor, and Brogden II teaches the second model is trained to identify contamination (see discussion per claim 2 above).
Regarding claim 4, the combined teaching of Brogden, Uemoto, Tanaka, Amthor, Brogden II, and Wikipedia may fail to explicitly disclose the second model is a segmentation model. However, Amthor teaches that non-semantic and semantic segmentation models can utilized to accomplish the intended operation of segmenting the image and separating background from identified elements (see Amthor, e.g. [0094]). Under the broadest reasonable interpretation of the claims, the claimed second model may be read as additionally including the second segmentation model of Amthor.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James Choi whose telephone number is (571) 272 – 2689. The examiner can normally be reached on 9:30 am – 6:00 pm M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Georgia Epps can be reached on (571) 272 – 2328. The fax phone number for the organization where this application or proceeding is assigned is (571) 273 – 8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES CHOI/Examiner, Art Unit 2878
Prosecution Insights