DETAILED ACTION
This communication is in response to the Application filed on May 13, 2024.
Claims 1 - 20 are pending and have been examined.
Claims 1, 9 and 17 are independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on May 13, 2024 and February 5, 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings filed on May 13, 2024 have been accepted and considered by the Examiner.
Nonstatutory Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1 - 2, 4 - 5, 8, 9, 12, 16 - 17, and 20 of the instant Application are provisionally rejected under the judicially created doctrine of obviousness-type double patenting as being unpatentable over claims 1 - 20 of copending application 18/747764 (hereinafter ‘764).
Regarding independent claims 1, 9, and 17, the conflicting claims are not identical to corresponding claims 4, 12 and 17 of the copending application ‘764 because the claims of copending application ‘764 require the additional limitation, not required by claims 1, 9 and 17 of the instant Application. However, the conflicting claims are not patentably distinct from each other because: (1) claims 1, 9 and 17 of the instant Application and claims 4, 12 and 17 of the copending application recite common subject matter, and (2) whereby the elements of claims 1 , 9 and 17 of instant Application are fully anticipated by claims 4, 12 and 17 of the copending application, and anticipation is “the ultimate or epitome of obviousness” (In re Kalm, 154 USPQ 10 (CCPA 1967), also In re Daily, 178 USPQ 293 (CCPA 1973) and In re Pearson, 181 USPQ 641 (CCPA 1974)). This is a provisional nonstatutory double patenting rejection.
Dependent claims 2, 4 - 5, 8, 12, 16, and 20 are also similarly analyzed and rejected over claims 1 - 20 of the copending application ‘764.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claims 1 - 2, 6, 9 - 10, 14, and 17 of the instant Application are provisionally rejected under the judicially created doctrine of obviousness-type double patenting as being unpatentable over claims 1 - 20 of copending application 18/747753 (hereinafter ‘753).
Regarding independent claims 1, 9, and 17, the conflicting claims are not identical to corresponding claims 1, 9 and 17 of the copending application because the claims of copending application ‘753 require the additional limitation, not required by claims 1, 9 and 17 of the instant Application. However, the conflicting claims are not patentably distinct from each other because: (1) claims 1, 9 and 17 of the instant Application and claims 1, 9 and 17 of the copending application recite common subject matter, and (2) whereby the elements of claims 1 , 9 and 17 of instant Application are fully anticipated by claims 1, 9 and 17 of the copending application, and anticipation is “the ultimate or epitome of obviousness” (In re Kalm, 154 USPQ 10 (CCPA 1967), also In re Daily, 178 USPQ 293 (CCPA 1973) and In re Pearson, 181 USPQ 641 (CCPA 1974)). This is a provisional nonstatutory double patenting rejection.
Dependent claims 2, 6, 10, and 14 are also similarly analyzed and rejected over claims 1 - 20 of the copending application ‘753.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1 - 2, 9 - 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ishikawa et al., (WO2025027758A1), hereinafter referred to as Ishikawa, in view of Amthor et al., (U.S. Patent Application Publication 2025/0102788), hereinafter referred to as Amthor.
Regarding Claims 1 and 9, Ishikawa teaches:
1. A system, comprising, and 9. A computer-implemented method, comprising:
a processor that executes computer-executable components stored in a non-transitory computer-readable memory, wherein the computer-executable components comprise: [Ishikawa, “The computer system 101 includes one or more processors 106 (including at least one of a CPU and a GPU), a memory 107, and a user interface 108.” Par. 0011; “In addition to the above, memory 107 (e.g., a non-transitory computer-readable storage medium) may store, for example: (a) an operating program executed by one or more processors included in semiconductor evaluation tool 102;” Par. 0019]
an access component that accesses a natural language instruction associated with a charged-particle microscope, wherein the natural language instruction requests or commands that the charged-particle microscope undergo a configurable settings adjustment or that the charged-particle microscope perform an automated task; [Ishikawa, “The semiconductor evaluation tool 102 is, for example, a CD-SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform based on the detection of secondary electrons and backscattered electrons emitted from a sample when the sample is irradiated with an electron beam.” Par. 0012; “The purpose of this disclosure is to input a desired operation (i.e., the claimed “access component that accesses a natural language instruction”) of a semiconductor device evaluation device (semiconductor evaluation tool 102 in FIG. 1) to the LLM, and to obtain an operation specification (evaluation recipe) that can realize the request as a response from the LLM. In other words, the purpose is to operate the semiconductor device evaluation device using natural language, etc.” Par. 0030; “The recipe generation module is configured to execute processes such as generating recipes and modifying generated recipes (i.e., the claimed “commands”) based on, for example, the output of the text creation module 111 (i.e., the claimed “configurable setting adjustment”) described below.” Par. 0020]
a state component that causes, in response to receipt of the natural language instruction, the charged-particle microscope to capture, according to a default microscopy protocol, an image or an energy spectrum of a specimen that is currently loaded on a stage of the charged-particle microscope; and a [Ishikawa, “The semiconductor evaluation tool 102 is, for example, a CD-SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image (i.e., the claimed “captures an image”) and a luminance signal waveform (i.e., the claimed “energy spectrum”) based on the detection of secondary electrons and backscattered electrons emitted from a sample (i.e., the claimed “specimen that is currently loaded on a stage”) when the sample (i.e., the claimed “specimen that is currently loaded on a stage”) is irradiated with an electron beam.” Par. 0012; “This allows the semiconductor LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”) that reflects the domain knowledge when an input statement relating to an evaluation apparatus or semiconductor device is given (i.e., the claimed “receipt of the natural language instruction”).” Par. 0043; “an observed image of a semiconductor device acquired by the evaluation device is abnormal, and measures to remedy the abnormality.” Par. 0050]
model component that executes a large language model on both the natural language instruction and the image or energy spectrum of the specimen, thereby yielding a natural language response that indicates how implementing the natural language instruction would affect the specimen. [Ishikawa, “The semiconductor evaluation tool 102 is, for example, a CD-SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image (i.e., the claimed “captures an image”) and a luminance signal waveform (i.e., the claimed “energy spectrum”) based on the detection of secondary electrons and backscattered electrons emitted from a sample (i.e., the claimed “specimen that is currently loaded on a stage”) when the sample (i.e., the claimed “specimen that is currently loaded on a stage”) is irradiated with an electron beam.” Par. 0012; “This allows the semiconductor LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”) that reflects the domain knowledge when an input statement relating to an evaluation apparatus or semiconductor device is given (i.e., the claimed “receipt of the natural language instruction”).” Par. 0043; “an observed image of a semiconductor device acquired by the evaluation device is abnormal, and measures to remedy the abnormality.” Par. 0050; “The sentence production module 234 (i.e. the claimed “model component”) uses the sentence production model 115 to output an output string (i.e., the claimed “natural language response that indicates how implementing the natural language instruction would affect the specimen”) as a response to an input string.” Par. 0026]
Ishikawa fails to explicitly teach both the natural language instruction and the image.
However, Amthor teaches:
model component that executes a large language model on both the natural language instruction and the image or energy spectrum of the specimen, thereby yielding a natural language response that indicates how implementing the natural language instruction would affect the specimen. [Amthor, “Holburn, D., et al., “Voice Control of the Scanning Electron Microscope Using a Low-Cost Virtual Assistant”, Microsc. Microanal. 27 (Suppl 1), 2021, doi: 10.1017/S1431927621009685. A user can give voice commands here such as “Autofocus”, “Capture image”, “Move x-axis by 100 steps”, whereupon the microscope implements these commands accordingly.” Par. 0004; “In principle, a structure or object depicted in microscope images and overview images can be any structure or object (i.e., the claimed “specimen”). Besides the sample itself—e.g., biological structures, electronic elements or rock fragments—it is also possible for a sample vessel, a sample carrier, a microscope component such as a sample stage (i.e., the claimed “specimen that is currently loaded on a stage”) or areas of the same to be depicted.” Par. 0140; “The large language model is a deep artificial neural network which receives (among other things) a text from a user as input (i.e., the claimed “natural language instruction”) and generates an output that specifies parameters for a subsequent image generation (i.e., the claimed “yielding a natural language response that indicates how implementing the natural language instruction would affect the specimen”).” Par. 0074; “For example, a user can tell the large language model (i.e., the claimed “natural language instruction”) whether a single cell of a particular type or a cell cluster of the sample should be imaged. The large language model uses this information to identify the appropriate magnification for capturing either a single cell or a cell cluster, while the overview image is used to navigate to an appropriate location where the desired cell(s) is (are) present. Imaging parameters (i.e., the claimed “thereby yielding a natural language response that indicates how implementing the natural language instruction would affect the specimen”) such as illumination intensity or fluorescence settings can be ascertained by the large language model as a function of the textual input (i.e., the claimed “natural language instruction”) and the overview image (i.e., the claimed “model component that executes a large language model on both the natural language instruction and the image of the specimen”) without the user having to specify the illumination intensity or fluorescence excitation or detection channels. This enables a high-quality imaging without requiring significant expertise of the user or a laborious performance of manual settings.” Par. 0032]
Ishikawa and Amthor pertain to artificial intelligence microscope systems and are analogous to the instant application. Accordingly, it would have been obvious to one of ordinary skill in the artificial intelligence microscope systems art to modify Ishikawa’s teachings of “LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”)” of a “SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform based on the detection of secondary electrons and backscattered electrons emitted from a sample when the sample is irradiated with an electron beam” (Ishikawa, Par. 0012, Par. 0043) with the explicit teachings of “large language model as a function of the textual input (i.e., the claimed “natural language instruction”) and the overview image (i.e., the claimed “model component that executes a large language model on both the natural language instruction and the image of the specimen”)” (Amthor, Par. 0032) taught by Amthor in order to “enable a high-quality imaging without requiring significant expertise of the user or a laborious performance of manual settings.” (Amthor, Par. 0032)
Regarding Claims 2 and 10, Ishikawa in view of Amthor has been discussed above. The combination further teaches:
wherein the computer-executable components further comprise: [Ishikawa, see mapping applied to claim 1]
a presenter component that: [Ishikawa, “The user interface 108 (i.e., the claimed “presenter component”) includes a display (not shown) and one or more input devices. The display can display image data output from the semiconductor evaluation tool 102, information output from the processor 106, and the like.” Par. 0011]
visibly renders the natural language response or a visual graphic associated with the natural language response on an electronic display associated with the charged-particle microscope; [Ishikawa, “The user interface 108 (i.e., the claimed “presenter component”) includes a display (not shown) and one or more input devices. The display can display image data output (i.e., the claimed “visibly renders a visual graphic associated with the natural language response on an electronic display associated with the charged-particle microscope”) from the semiconductor evaluation tool 102, information output from the processor 106 (i.e., the claimed “visibly renders the natural language response”), and the like.” Par. 0011]
Claims 3 - 5, 11 - 13, 17 - 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ishikawa in view of Amthor and Napolitano et al., (AU2017204359A1), hereinafter referred to as Napolitano.
Regarding Claims 3 and 11, Ishikawa in view of Amthor has been discussed above. The combination further teaches:
plain text that is typed into a graphical user-interface text field associated with the charged-particle microscope; [Ishikawa, “The user interface 108 (i.e., the claimed “graphical user interface”) includes a display (not shown) and one or more input devices (i.e. the claimed “plain text that is typed into a graphical user interface”). The display can display image data output from the semiconductor evaluation tool 102, information output from the processor 106, and the like.” Par. 0011; “The present disclosure has been made in consideration of the above-described problems, and aims to provide a technology that enables instructions for an evaluation device that evaluates semiconductor devices to be obtained from a language model by inputting a string of characters (i.e., the claimed “plain text that is typed”) into the language model.” Par. 0006; “This allows instructions to be given to the evaluation device by means of a character string via the LLM. In other words, the evaluation device can be operated by a string of characters.” Par. 0044]
The combination fails to explicitly teach text field.
However, Napolitano teaches:
plain text that is typed into a graphical user-interface text field associated with the charged-particle microscope; [Napolitano, “At block 560 of process 500, a search field configured to receive typed search inputs (i.e., the claimed “plain text that is typed”) can be displayed. For example, as shown in FIG. 6L, search field 644 (i.e., the claimed “text field”) can be displayed on the displayed unit (i.e., the claimed “graphical user interface”).” Par. 0180]
Ishikawa, Amthor and Napolitano pertain to artificial intelligence systems and are analogous to the instant application. Accordingly, it would have been obvious to one of ordinary skill in the artificial intelligence systems art to modify Ishikawa’s teachings of “LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”)” of a “SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform based on the detection of secondary electrons and backscattered electrons emitted from a sample when the sample is irradiated with an electron beam” (Ishikawa, Par. 0012, Par. 0043) with the explicit teachings of “large language model as a function of the textual input (i.e., the claimed “natural language instruction”) and the overview image (i.e., the claimed “model component that executes a large language model on both the natural language instruction and the image of the specimen”)” (Amthor, Par. 0032) taught by Amthor and the explicit teachings of “field displayed on a display unit” (Napolitano, Par. 0180) taught by Napolitano in order to “enable a high-quality imaging without requiring significant expertise of the user or a laborious performance of manual settings” (Amthor, Par. 0032) and enable “plurality of exemplary natural language requests can be displayed” (Napolitano, Par. 0006).
Regarding Claims 4 and 12, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
wherein the natural language response indicates that implementing the natural language instruction would harm or charge the specimen, and [Amthor, “A response A from the user U to the follow-up query Q is processed by the large language model LLM in order to either use the adjusted microscope settings 40B to capture the new microscope image 50B, or to modify the adjusted microscope settings 40B again based on the response A from the user U. For example, if the user responds that a photodamaging of the sample 10 is unacceptable (i.e., the claimed “harm”), the large language model LLM can (potentially after a further follow-up query Q and associated response A (i.e., the claimed “natural language response indicates that implementing the natural language instruction”)) increase the illumination duration and measurement duration in order to thereby achieve a better visibility of the particular cell type without increasing the illumination intensity. Alternatively, the large language model LLM can switch to an objective with a higher magnification and capture a plurality of laterally offset microscope images that are stitched together to form one image (image stitching), which can also achieve a better visibility of the particular cell type without increasing the illumination intensity.” Par. 0170]
wherein the natural language response further indicates that the charged-particle microscope should undergo an alternative configurable settings adjustment or that the charged-particle microscope should perform an alternative automated task. [Amthor, “A response A from the user U to the follow-up query Q is processed by the large language model LLM in order to either use the adjusted microscope settings 40B to capture the new microscope image 50B, or to modify the adjusted microscope settings 40B again based on the response A from the user U. For example, if the user responds that a photodamaging of the sample 10 is unacceptable (i.e., the claimed “harm”), the large language model LLM can (potentially after a further follow-up query Q and associated response A (i.e., the claimed “natural language response indicates that implementing the natural language instruction”)) increase the illumination duration and measurement duration in order to thereby achieve a better visibility of the particular cell type without increasing the illumination intensity. Alternatively, the large language model LLM can switch to an objective with a higher magnification (i.e. ,the claimed “alternative configurable settings”/ “alternative automated task”)) and capture a plurality of laterally offset microscope images that are stitched together to form one image (image stitching), which can also achieve a better visibility of the particular cell type without increasing the illumination intensity.” Par. 0170]
Regarding Claims 5 and 13, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
wherein the computer-executable components further comprise: [Ishikawa, see mapping applied to claim 1]
a presenter component that causes the charged-particle microscope to undergo the alternative configurable settings adjustment or to perform the alternative automated task. [Ishikawa, see mapping applied to claims 1 and 3; Napolitano, see mapping applied to claim 3; Amthor, see mapping applied to claim 4]
Regarding Claim 17, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
17. A computer program product for facilitating large language model assistance for charged-particle microscope operation, the computer program product comprising a non-transitory computer-readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: [Ishikawa, see mapping applied to claim 1; “In addition to the above, memory 107 (e.g., a non-transitory computer-readable storage medium) may store, for example: (a) an operating program (i.e., the claimed “computer program product”) executed by one or more processors included in semiconductor evaluation tool 102;” Par. 0019]
access a plain text command provided by a user of a scanning electron microscope, [Ishikawa, see mapping applied to claim 1, 3]
wherein the plain text command requests that the scanning electron microscope perform a specified microscopy action; [Ishikawa, see mapping applied to claims 1, 3; Napolitano, see mapping applied to claim 3; Napolitano, “In some embodiments, the task performed can depend on the nature of die user request (i.e., the claimed “perform specified action”) and the content that is displayed while the user input of a second input type is detected.” Par. 0031; Amthor, “A response A from the user U to the follow-up query Q is processed by the large language model LLM in order to either use the adjusted microscope settings 40B to capture the new microscope image (i.e., the claimed “perform a specified microscopy action”) 50B, or to modify the adjusted microscope settings 40B again based on the response A from the user U.” Par. 0170]
in response to receipt of the plain text command, cause the scanning electron microscope to capture, via a default microscopy protocol, an image or energy spectrum of a specimen that is currently loaded on a stage of the scanning electron microscope; [Ishikawa, see mapping applied to claims 1, 3; Amthor, see mapping applied to claim 1; Napolitano, see mapping applied to claim 3; Napolitano, “In some embodiments, the task performed can depend on the nature of die user request (i.e., the claimed “perform specified action”) and the content that is displayed while the user input of a second input type is detected.” Par. 0031; Amthor, “A response A from the user U to the follow-up query Q is processed by the large language model LLM in order to either use the adjusted microscope settings 40B to capture the new microscope image (i.e., the claimed “perform a specified microscopy action”) 50B, or to modify the adjusted microscope settings 40B again based on the response A from the user U.” Par. 0170]
execute a large language model on both the plain text command and the image or energy spectrum of the specimen, [Ishikawa, see mapping applied to claim 1; Amthor, see mapping applied to claim 1]
wherein the large language model produces as output a plain text response that indicates whether the specified microscopy action would damage the specimen; and [Ishikawa, see mapping applied to claims 1, 3; Amthor, see mapping applied to claim 1; Napolitano, see mapping applied to claim 3; Napolitano, “In some embodiments, the task performed can depend on the nature of die user request (i.e., the claimed “perform specified action”) and the content that is displayed while the user input of a second input type is detected.” Par. 0031; Amthor, “A response A from the user U to the follow-up query Q is processed by the large language model LLM in order to either use the adjusted microscope settings 40B to capture the new microscope image (i.e., the claimed “perform a specified microscopy action”) 50B, or to modify the adjusted microscope settings 40B again based on the response A from the user U.” Par. 0170]
visibly or audibly render the plain text response on an electronic display or on an electronic speaker associated with the scanning electron microscope. [Ishikawa, see mapping applied to claims 1 - 2; Amthor, see mapping applied to claim 1]
Regarding Claim 18, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
wherein the plain text response indicates that the specified microscopy action would damage the specimen, and [Ishikawa, see mapping applied to claims 1, 3, 17; Napolitano, see mapping applied to claim 3, 17; Amthor, see mapping applied to claim 1, 4, 17]
wherein the plain text response further indicates that such damage is avoidable by an alternative microscopy action. [Ishikawa, see mapping applied to claims 1, 3, 17; Napolitano, see mapping applied to claim 3, 17; Amthor, see mapping applied to claim 1, 4, 17]
Regarding Claim 19, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
wherein the program instructions are further executable to cause the processor to: [Ishikawa, see mapping applied to claim 1]
instruct the scanning electron microscope to perform the alternative microscopy action. [Ishikawa, see mapping applied to claims 1, 3, 17 - 18; Napolitano, see mapping applied to claim 3, 17 - 18; Amthor, see mapping applied to claim 1, 4, 17 - 18]
Claims 6 - 7, 14 - 15 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ishikawa in view of Amthor and Larson et al., (U.S. Patent Application Publication 2025/0328565), hereinafter referred to as Larson.
Regarding Claims 6 and 14, Ishikawa in view of Amthor has been discussed above. The combination further teaches:
wherein the computer-executable components further comprise: [Ishikawa, see mapping applied to claim 1]
one or more documents that are relevant to the natural language instruction and to the image or energy spectrum of the specimen, and [Ishikawa, “Fine tuning can be performed for domain knowledge (i.e., the claimed “documents”) related to the evaluation device by having the general-purpose LLM 210 learn document data (i.e., the claimed “one or more documents”),” Par. 0032; “The document creation system 200 includes: (a) an equipment supplier-side computer system 220 that performs fine tuning of the generic LLM 210 mainly based on domain knowledge (i.e., the claimed “documents”) of the evaluation equipment;” Par. 0022; “The semiconductor evaluation tool 102 is, for example, a CD-SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image (i.e., the claimed “captures an image”) and a luminance signal waveform (i.e., the claimed “energy spectrum”) based on the detection of secondary electrons and backscattered electrons emitted from a sample (i.e., the claimed “specimen that is currently loaded on a stage”) when the sample (i.e., the claimed “specimen that is currently loaded on a stage”) is irradiated with an electron beam.” Par. 0012]
wherein the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and the one or more documents. [Ishikawa, “Fine tuning can be performed for domain knowledge (i.e., the claimed “documents”) related to the evaluation device by having the general-purpose LLM 210 (i.e., the claimed “large language model”) learn document data (i.e., the claimed “one or more documents”),” Par. 0032; “The document creation system 200 includes: (a) an equipment supplier-side computer system 220 that performs fine tuning of the generic LLM 210 (i.e., the claimed “large language model”) mainly based on domain knowledge (i.e., the claimed “documents”) of the evaluation equipment;” Par. 0022; “The purpose of this disclosure is to input a desired operation (i.e., the claimed “natural language instruction”) of a semiconductor device evaluation device (semiconductor evaluation tool 102 in FIG. 1) to the LLM (i.e., the claimed “large language model”), and to obtain an operation specification (evaluation recipe) that can realize the request as a response from the LLM. In other words, the purpose is to operate the semiconductor device evaluation device using natural language, etc.” Par. 0030; “The semiconductor evaluation tool 102 is, for example, a CD-SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform (i.e., the claimed “energy spectrum”) based on the detection of secondary electrons and backscattered electrons emitted from a sample (i.e., the claimed “specimen” when the sample (i.e., the claimed “specimen”) is irradiated with an electron beam.” Par. 0012]
The combination fails to teach a context component that identifies, via an embedding search of a document repository.
However, Larson teaches:
a context component that identifies, via an embedding search of a document repository, [Larson, “A scientific instrument (e.g., mass spectrometer, charged-particle microscope) can be any suitable computerized device that can capture or generate electronic measurements in a scientific, laboratory, research, or clinical operational context (e.g., that can capture or generate spectroscopic images or composition spectra).” Par. 0026; “a repository or database of document-graphs, each document-graph comprising respective context-tagged text blocks; composition of adjacent context-tagged text blocks via iterative graph-walking and embedding-change comparison;” Par. 0036; “In various instances, such searching can be accomplished via embedding techniques (i.e., the claimed “embedding search”) or via keyword techniques. In various cases, when a relevant (or potentially-relevant) context-tagged text block is found,” Par. 0038; “In various embodiments, the search component of the computerized tool (i.e., the claimed “context component”) can electronically store, maintain, control, or otherwise access a document-graph repository. In various aspects, the search component can electronically leverage the document-graph repository so as to identify a plurality of context-tagged text blocks that are substantively relevant to the plain text question.” Par. 0050]
Ishikawa, Amthor and Larson pertain to artificial intelligence systems and are analogous to the instant application. Accordingly, it would have been obvious to one of ordinary skill in the artificial intelligence systems art to modify Ishikawa’s teachings of “LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”)” of a “SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform based on the detection of secondary electrons and backscattered electrons emitted from a sample when the sample is irradiated with an electron beam” (Ishikawa, Par. 0012, Par. 0043) with the explicit teachings of “large language model as a function of the textual input (i.e., the claimed “natural language instruction”) and the overview image (i.e., the claimed “model component that executes a large language model on both the natural language instruction and the image of the specimen”)” (Amthor, Par. 0032) taught by Amthor and the teachings of “searching can be accomplished via embedding techniques (i.e., the claimed “embedding search”)” (Larson, Par. 0038) taught by Larson in order to “enable a high-quality imaging without requiring significant expertise of the user or a laborious performance of manual settings” (Amthor, Par. 0032) and automatically answer “inquiries regarding how the scientific instrument should be operated, maintained, serviced, or troubleshot.” (Larson, Par. 0027)
Regarding Claims 7 and 15, Ishikawa in view of Amthor and Larson has been discussed above. The combination further teaches:
wherein the computer-executable components further comprise: [Ishikawa, see mapping applied to claim 1]
a context component that executes one or more available deep learning models on the image or energy spectrum of the specimen, [Larson, “In various embodiments, there can be a LLM. In various aspects, the LLM can exhibit any suitable deep learning internal architecture (i.e., the claimed “deep learning model”).” Par. 0044; “A scientific instrument (e.g., mass spectrometer, charged-particle microscope) can be any suitable computerized device that can capture or generate electronic measurements in a scientific, laboratory, research, or clinical operational context (e.g., that can capture or generate spectroscopic images or composition spectra).” Par. 0026; “In various instances, such searching can be accomplished via embedding techniques (i.e., the claimed “embedding search”) or via keyword techniques. In various cases, when a relevant (or potentially-relevant) context-tagged text block is found,” Par. 0038; “In various embodiments, the search component of the computerized tool (i.e., the claimed “context component”) can electronically store, maintain, control, or otherwise access a document-graph repository. In various aspects, the search component can electronically leverage the document-graph repository so as to identify a plurality of context-tagged text blocks that are substantively relevant to the plain text question.” Par. 0050]
thereby yielding one or more inferencing task results, [Larson, “This can ultimately cause the trainable internal parameters of the artificial intelligence model (e.g., of the LLM 306, of the text-to-graph neural network 806, of the named entity recognition neural network 812, of the re-ranker 1402) to become iteratively optimized for accurately performing its inferencing task (e.g., text synthesis, graph synthesis, named entity extraction, relevance score computation) (i.e., the claimed “inferencing task results”).” Par. 0219]
wherein the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and the one or more inferencing task results. [Ishikawa, see mapping applied to claim 1; Larson, see mapping applied to claim 6; Larson, “This can ultimately cause the trainable internal parameters of the artificial intelligence model (e.g., of the LLM 306, of the text-to-graph neural network 806, of the named entity recognition neural network 812, of the re-ranker 1402) to become iteratively optimized for accurately performing its inferencing task (e.g., text synthesis, graph synthesis, named entity extraction, relevance score computation) (i.e., the claimed “inferencing task results”).” Par. 0219]
Claims 8, 16, 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ishikawa in view of Amthor, Napolitano, Weber et al., ("Calibrating Coordinate System Alignment in a Scanning Transmission Electron Microscope using a Digital Twin," arXiv:2403.08538, 2024), hereinafter referred to as Weber, and Batlkhagva et al., ("Digital Twin: Virtual Hardware Simulator for a Transmission electron microscope," Eindhoven University of Technology, PDEng Report, 2020).
Regarding Claims 8 and 16, Ishikawa in view of Amthor and Napolitano has been discussed above. The combination further teaches:
wherein the charged-particle microscope is synchronized with a digital twin, and wherein: [Ishikawa, see mapping applied to claim 1]
the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and a current health status of the charged-particle microscope as indicated by the digital twin; [Ishikawa, see mapping applied to claim 1]
or
the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and one or more simulation results regarding the charged-particle microscope produced by the digital twin. [Ishikawa, see mapping applied to claim 1]
Ishikawa in view of Amthor and Napolitano fails to teach digital twin.
However, Weber teaches:
wherein the charged-particle microscope is synchronized with a digital twin, and wherein: [Ishikawa, see mapping applied to claim 1; Weber, “4D scanning transmission electron microscopy (STEM) (i.e., the claimed “charged-particle microscope)” Pg. 1; “a digital twin is used to match a set of models and their parameters with the action of a real-world instrument.” Pg. 1; “It uses automated data processing and a digital twin of the microscope (i.e., the claimed “charged-particle microscope”),” Pg. 2]
the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and one or more simulation results regarding the charged-particle microscope produced by the digital twin. [Ishikawa, see mapping applied to claim 1; Weber, “It uses automated data processing and a digital twin of the microscope to superimpose all shadow images in an over- or underfocused 4D STEM dataset. If the transformation by the digital twin matches the actual transformation by the microscope, a sharp image is obtained.” Pg. 2; “simulated optical elements such as deflectors and lenses to yield results equivalent to the abstract model.” Pg. 3; “a digital twin is used to match a set of models and their parameters with the action of a real-world instrument.” Pg. 1; “It uses automated data processing and a digital twin of the microscope (i.e., the claimed “charged-particle microscope”),” Pg. 2]
Ishikawa in view of Amthor, Napolitano and Weber fails to teach current health status.
However, Batlkhagva teaches:
the large language model receives as input the natural language instruction, the image or energy spectrum of the specimen, and a current health status of the charged-particle microscope as indicated by the digital twin; [Ishikawa, see mapping applied to claim 1; Batlkhagva, “The second conclusion is that the concept of the digital twin is beneficial for product maintenance (i.e., the claimed “current health status”) by visualizing the sensor data of the real system. From the visualization, engineers are able to inspect the real-time motion behavior of physical hardware (i.e., the claimed “current health status”) to diagnose problems.” Pg. viii]
Ishikawa, Amthor, Napolitano, Weber and Batlkhagva and pertain to artificial intelligence microscope systems and are analogous to the instant application. Accordingly, it would have been obvious to one of ordinary skill in the artificial intelligence microscope systems art to modify Ishikawa’s teachings of “LLM 232 to output a response (i.e., the claimed “response to receipt of the natural language instruction”)” of a “SEM (Critical Dimension-Scanning Electron Microscope (i.e., the claimed “charged-particle microscope)), which is a device that generates an image and a luminance signal waveform based on the detection of secondary electrons and backscattered electrons emitted from a sample when the sample is irradiated with an electron beam” (Ishikawa, Par. 0012, Par. 0043) with the explicit teachings of “large language model as a function of the textual input (i.e., the claimed “natural language instruction”) and the overview image (i.e., the claimed “model component that executes a large language model on both the natural language instruction and the image of the specimen”)” (Amthor, Par. 0032) taught by Amthor, the explicit teachings of “field displayed on a display unit” (Napolitano, Par. 0180) taught by Napolitano, the teachings of “digital twin is used to match a set of models and their parameters with the action of a real-world instrument” (Weber, Pg. 1) taught by Weber, and the teachings “product maintenance”/ “real-time motion behavior of physical hardware”/ current health status (Batlkhagva, Pg. viii) taught by Batlkhagva in order to “enable a high-quality imaging without requiring significant expertise of the user or a laborious performance of manual settings” (Amthor, Par. 0032), enable “plurality of exemplary natural language requests can be displayed” (Napolitano, Par. 0006), “eliminate error sources” with a “digital twin that matches the actual transformation by the microscope” (Weber, Pg. 2), and enable “service team to inspect problems when maintenance is required” and “as a result, problem diagnosis is faster when TEM misbehaves” (Batlkhagva, Pg. viii).
Regarding Claim 20, Ishikawa in view of Amthor, Napolitano, Weber and Batlkhagva has been discussed above. The combination further teaches:
wherein the large language model receives as input the plain text command, the image or energy spectrum of the specimen, and one or more simulation results produced by a digital twin that is synchronized with the scanning electron microscope. [Ishikawa, see mapping applied to claim 8; Weber, see mapping applied to claim 8; Batlkhagva, see mapping applied to claim 8; Weber, “4D scanning transmission electron microscopy (STEM) (i.e., the claimed “scanning electron microscope)” Pg. 1]
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chandran et al., (U.S. Patent Application Publication 2025/0139342) teaches digital twins.
Lu et al., (U.S. Patent Application Publication 2024/0272926) teaches digital twins.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EUNICE LEE whose telephone number is 571-272-1886. The examiner can normally be reached M-F 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached on 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EUNICE LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/ Supervisory Patent Examiner, Art Unit 2656