DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Examiner acknowledges that this is a continuation application of International Application No. PCT/CN2021/108514, filed on July 26, 2021.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 12/28/2023 has been reviewed and the listed references have been considered.
Drawings
The 7-page drawings have been considered and placed on record in the file.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “deformation module” and “neural radiance module”.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, these are being interpreted to cover the corresponding structures described in the applicant’s drawings: diagram depicted in Fig. 2B, and applicant’s specification: ¶0024: “machine learning model can comprise a space-time deformation module and a neural radiance module”; and ¶0038: “the processor 402 can encode a machine learning model” as performing the claimed functions, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. For example, Claim 12 recites “The non-transitory computing medium of claim 11”, wherein, claim 11 recites “A non-transitory computer-readable media”. Therefore, there is insufficient antecedent basis for “computing medium” in the claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-9, 11-13 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Martin Brualla et al. (US 2024/0005590 A1, hereinafter “Martin”) in view of Guler et al. (US 2022/0358770 A1).
Regarding claim 1, Martin teaches, A computer-implemented method comprising: (Martin, ¶0075: “The computer program product may also contain instructions that, when executed, perform one or more methods”) obtaining, by a computing system, a plurality of images (Martin, ¶0004: “a method can include acquiring image data representing a plurality of images”) of an object from a plurality of orientations (Martin, ¶0018: “In contrast to conventional NeRFs, the technical solution first expresses the positions of the subjects from various perspectives in an observation frame”) at a plurality of times; (Martin, ¶0024: “NeRF is a continuous, volumetric representation”) encoding, by the computing system, a machine learning model (Martin, ¶0087: “a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language”) to represent a continuous density field of the object, wherein the continuous density field maps a spatial coordinate to a density value, (Martin, ¶0015: “one represents a static scene as a continuous five-dimensional function that outputs the radiance emitted in each direction (θ, φ) at each point (x, y, z) in space, and a density at each point which acts like a differential opacity controlling how much radiance is accumulated by a ray passing through (x, y, z)”) and the machine learning model comprises: a deformation module configured to deform the spatial coordinate in accordance with a timestamp (Martin, ¶0027: “An observation-to-canonical deformation is employed for every frame i∈{1, . . . , n}, where n is the number of observed frames. This defines a mapping T.sub.i:x.fwdarw.x′ that maps all observation-space coordinates x to a canonical-space coordinate x′. In practice, the deformation fields are modeled for all time steps”) and a trained deformation weight (Martin, ¶0023: “the periodic function of the positional encoding is multiplied by a weight indicating whether a training iteration includes a particular frequency”) and to obtain a deformed spatial coordinate; (Martin, ¶0023: “deformation model includes applying a positional encoding to a position coordinate within the scene to produce a periodic function of position”) and a neural radiance module configured to derive the density value (Martin, ¶0005: “a deformable neural radiance field (D-NeRF)… the D-NeRF providing a mapping between the positions and viewing directions to a color and optical density at each position in the observation frame”) in accordance with the deformed spatial coordinate, the timestamp, a direction, and a trained radiance weight; (Martin, ¶0015: “continuous five-dimensional function that outputs the radiance emitted in each direction (θ, φ) at each point (x, y, z) in space, and a density at each point which acts like a differential opacity controlling how much radiance is accumulated by a ray passing through (x, y, z)”). However, Martin does not explicitly teach, training, by the computing system, the machine learning model using the plurality of images to obtain a trained machine learning model; and constructing, by the computing system, a three-dimensional structure of the object based on the trained machine learning model.
In an analogous field of endeavor, Guler teaches, training, by the computing system, the machine learning model using the plurality of images to obtain a trained machine learning model; and constructing, by the computing system, a three-dimensional structure of the object based on the trained machine learning model.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin using the teachings of Guler to introduce a trained machine learning model. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of automatically generating 3D structures from 2D images. Therefore, it would have been obvious to combine the analogous arts Martin and Guler to obtain the invention in claim 1.
Regarding claim 2, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein each image of the plurality of images comprises an image identification, and the image identification is encoded into a high dimension feature using positional encoding. (Martin, ¶0048: “A core component of the NeRF architecture is positional encoding… in a high dimensional space using a set of sine and cosine functions of increasing frequencies”).
Regarding claim 3, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein the spatial coordinate, the direction, (Martin, ¶0024: “maps a 3D position x=(x, y, z) and viewing direction d=(ϕ, θ) to an RGB color c=(r, g, b) and density σ. Coupled with volume rendering techniques, NeRFs can represent scenes with photo-realistic quality”) and the timestamp are encoded (Martin, ¶0023: “applying a positional encoding to a position coordinate within the scene to produce a periodic function of position”) into a high dimension feature using positional encoding. (Martin, ¶0048: “positional encoding… in a high dimensional space using a set of sine and cosine functions of increasing frequencies”).
Regarding claim 5, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein the deformation module comprises a first multi-layer perceptron (MLP). (Martin, ¶0022: “the deformation model includes a multilayer perceptron (MLP) within a neural network”).
Regarding claim 6, Martin in view of Guler teaches, The computer-implemented method of claim 5, wherein the first MLP comprises an 8-layer MLP (Martin, ¶0039: “deformation field MLP in the context of the technical solution describe herein has six layers”) with a skip connection at a fourth layer. (Martin, ¶0039: “there is a skip connection at the fourth layer”).
Regarding claim 7, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein the neural radiance module comprises a second multi-layer perceptron (MLP). (Martin, ¶0018: “. A NeRF is then derived from positions and casted ray directions in the canonical frame using another MLP”)
Regarding claim 8, Martin in view of Guler teaches, The computer-implemented method of claim 7, wherein the second MLP comprises an 8-layer multi-layer perceptron (MLP) (Martin, ¶0055: “NeRF MLP in the context of the technical solution describe herein has six layers (one input, one output”) with a skip connection at a fourth layer. (Martin, ¶0055: “there is a skip connection at the fourth layer”).
Regarding claim 9, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein training the machine learning model using the plurality of images comprises: partitioning the plurality of images into a plurality of bins; selecting a plurality of first sample images from the plurality of bins, wherein each of the plurality of first sample images is selected from a bin of the plurality of bins; (Martin, ¶0061: “we use a stratified sampling approach where we partition [t.sub.n, t.sub.f] into M evenly-spaced bins and then draw one sample uniformly at random from within each bin”) and training the machine learning model using the plurality of first sample images. (Martin, ¶0061: “stratified sampling enables a representation of a continuous scene because the stratified sampling results in the MLP being evaluated at continuous positions over the course of optimization”).
Regarding claim 11, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 1. Therefore, the recited instructions of the computer-readable media of claim 11 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 1. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim. Additionally, Martin teaches, A non-transitory computer-readable media of a computing system storing instructions, (Martin, ¶0081: “The memory 564 stores information within the computing device 550. The memory 564 can be implemented as one or more of a computer-readable medium”) wherein when the instructions are executed by one or more processors of the computing system, the computing system performs a method comprising: (Martin, ¶0079: “The processor 552 can execute instructions within the computing device 450, including instructions stored in the memory 564”).
Regarding claim 12, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 2. Therefore, the recited instructions of the computer-readable media of claim 12 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 2. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 13, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 3. Therefore, the recited instructions of the computer-readable media of claim 13 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 3. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 15, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 5. Therefore, the recited instructions of the computer-readable media of claim 15 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 5. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 16, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 6. Therefore, the recited instructions of the computer-readable media of claim 16 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 6. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 17, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 7. Therefore, the recited instructions of the computer-readable media of claim 17 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 7. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 18, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 8. Therefore, the recited instructions of the computer-readable media of claim 18 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 8. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Regarding claim 19, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 9. Therefore, the recited instructions of the computer-readable media of claim 19 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 9. Additionally, the rationale and motivation to combine Martin and Guler presented in rejection of claim 1, apply to this claim.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Martin Brualla et al. (US 2024/0005590 A1, hereinafter “Martin”) in view of Guler et al. (US 2022/0358770 A1) and in further view of Zandbergen (US 2020/0141846 A1).
Regarding claim 4, Martin in view of Guler teaches, The computer-implemented method of claim 1, wherein obtaining the plurality of images of the object from the plurality of orientations at the plurality of times comprises. However, the combination of Martin and Guler does not explicitly teach, obtaining a plurality of cryo-ET images by mechanically tilting the object at different angles.
In an analogous field of endeavor, Zandbergen teaches, obtaining a plurality of cryo-ET images by mechanically tilting the object at different angles. (Zandbergen, ¶0023: “For single particle recording no significant tilt is required. For tomographic cryo TEM one typically requires a large tilt over which many images of the sample object are taken”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin in view of Guler using the teachings of Zandbergen to introduce capturing images of an object by tilting the object. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of automatically generating 3D structures from 2D images of different orientations of the object. Therefore, it would have been obvious to combine the analogous arts Martin, Guler and Zandbergen to obtain the invention in claim 4.
Regarding claim 14, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 4. Therefore, the recited instructions of the computer-readable media of claim 14 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 4. Additionally, the rationale and motivation to combine Martin, Guler and Zandbergen presented in rejection of claim 4, apply to this claim.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Martin Brualla et al. (US 2024/0005590 A1, hereinafter “Martin”) in view of Guler et al. (US 2022/0358770 A1) and in further view of Chen et al. (US 2021/0174144 A1).
Regarding claim 10, Martin in view of Guler teaches, The computer-implemented method of claim 9, further comprising. However, the combination of Martin and Guler does not explicitly teach, producing, by the computing system, a piecewise-constant probability distribution function (PDF) for the plurality of images based on the machine learning model; selecting, by the computing system, a plurality of second sample images from the plurality of images in accordance with the piecewise-constant PDF; and further training, by the computing system, the machine learning model using the plurality of second sample images.
In an analogous field of endeavor, Chen teaches, producing, by the computing system, a piecewise-constant probability distribution function (PDF) for the plurality of images based on the machine learning model; (Chen, ¶0006: “the second model includes a probability function corresponding to feature data of an input sample”) selecting, by the computing system, a plurality of second sample images from the plurality of images in accordance with the piecewise-constant PDF; (Chen, ¶0006: “calculates a probability of selecting the input sample as a training sample of the first model based on the probability function, and outputs a corresponding output value based on the probability”) and further training, by the computing system, the machine learning model using the plurality of second sample images. (Chen, ¶0006: “training the second model by using a policy gradient algorithm based on the feature data of the at least one second sample, a probability function corresponding to each feature data in the second model”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Martin in view of Guler using the teachings of Chen to introduce a probability function. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of selecting a sample of training images based on quantified likelihoods. Therefore, it would have been obvious to combine the analogous arts Martin, Guler and Chen to obtain the invention in claim 10.
Regarding claim 20, it recites a computer-readable media instructions corresponding to the steps of the method recited in claim 10. Therefore, the recited instructions of the computer-readable media of claim 20 are mapped to the proposed combination in the same manner as the corresponding steps of the method claim 10. Additionally, the rationale and motivation to combine Martin, Guler and Chen presented in rejection of claim 10, apply to this claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEHRAZUL ISLAM whose telephone number is (571)270-0489. The examiner can normally be reached Monday-Friday: 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Saini Amandeep can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEHRAZUL ISLAM/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662