DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/13/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Allowable Subject Matter
Claim 10 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-9, 11-13, 15-17, 19-20 is/are rejected under 35 U.S.C. 102(a) as being taught by Metaxas et al. (U.S. Patent Publication No. 2026/0080553-A1, hereinafter “Metaxas”).
Regarding claim 1, Metaxas teaches: A scanner for image reconstruction of a structure of a scene, comprising:a memory configured to store instructions; and at least one processor configured to execute the instructions to cause the scanner to: ([0031], "The one or more output image(s) 22 is/are digitally propagated to one or more user-defined or automatically generated surface(s) 30 (seen in FIG. 1B). The system 2 includes a computing device 100 that contains one or more processors 102 therein and image processing software 104 that incorporates the trained recurrent neural network 10. The computing device 100 may include, as explained herein, a personal computer, laptop, tablet PC, remote server, application-specific integrated circuit (ASIC), or the like, although other computing devices 100 may be used (e.g., devices that incorporate one or more graphic processing units (GPUs)).")
collect measurements of intensities of a wave over a period of time, wherein the intensities of the wave are modified by propagation of the wave in the scene; ([0034], "In some embodiments, a series or time sequence of output images 22 are generated by the trained recurrent neural network 10, e.g., a time-lapse video clip or movie of the sample 12 or objects therein. The trained recurrent neural network 10 receives one or more microscopy input image(s) 20 (e.g., multiple images taken at different times) of the sample 12 (e.g., fluorescence microscopy images) obtained by the microscope device 110."; [0074], "The images of C. elegans were captured by an inverted scanning microscope (TCS SP8, Leica Microsystems), using a 63×/1.4 NA objective lens (HC PL APO 63×/1.4 NA oil CS2, Leica Microsystems) and a FITC filter set (excitation/emission wavelengths: 495 nm/519 nm), resulting in a DoF about 0.4 μm. A monochrome scientific CMOS camera (Leica DFC9000GTC-VSC08298) was used for imaging where each image has 1024×1024 pixels and 12-bit dynamic range. For each FOV, 91 images with 0.2 μm axial spacing were recorded. A total of 100 FOVs were captured and exclusively divided into training, validation and testing datasets at the ratio of 41:8:1, respectively, where the testing dataset was strictly captured on distinct worms that were not used in training dataset.")
collect depth information indicative of the structure of the scene at different values of depth of the scene, wherein the different values of depth correlate with different time segments forming the period of time; process the measurements with a guided recurrent neural network to sequentially learn features of the structure of the scene using the depth information as a guidance, ([0039], "In other embodiments, multiple user-defined or automatically generated surfaces 30 may be combined to create a volumetric (3D) image of the sample 12 using a plurality of output images 22 (FIG. 2A). Thus, a stack of output images 22 generated using the trained recurrent neural network 10 may be merged or combined to create a volumetric image of the sample 12. The volumetric image may also be generated as a function of time, e.g., a volumetric movie or time-lapse video clip that shows movement within the volume over time.")
wherein the depth information is aligned with the measurements according to the correlation between the depth values and the different time segments and render one or multiple images indicative of the features of the structure learned by the recurrent neural network. ([0040], " For example, the image processing software 104 may automatically generate one or more DPMs 26 that correct for one or more optical aberrations. This may include aberrations such as sample drift, tilt and spherical aberrations. Thus, the DPM(s) 26 may be automatically generated by an algorithm implemented in the image processing software 104. Such an algorithm, which may be implemented using a separate trained neural network or software, may operate by having an initial guess with a surface or DPM 26 that is input with an input image 20. The result of the network or software output is analyzed according to a metric (e.g., sharpness or contrast). The result is then used to generate a new surface represented by a different DPM 26 that is input with an input image 20 and analyzed as noted above until the result has converged on a satisfactory result (e.g., sufficient sharpness or contrast has been achieved or a maximum result obtained). ")
Regarding claim 3, Metaxas teaches: The scanner of claim 2, wherein the guided recurrent neural network includes a sequence of recurrent units that sequentially learn the features of the structure of the scene, wherein each of the recurrent units is associated with a time segment from the different time segments forming the period of time. ([0083], "In this work, the gated recurrent unit (GRU) is used as the recurrent unit, i.e., the RConv(⋅) layer in Eq. (2) updates h.sub.t, given the input x.sub.t")
Regarding claim 4, Metaxas teaches: The scanner of claim 3, wherein a recurrent unit of the sequence of recurrent units is configured to learn at least some features of the features of the structure of the scene based on an output of a previous iteration, a portion of the measurements collected over an associated time segment, and a quantized value of a bin mapped to the associated time segment. (Fig.2B; [0083], "In this work, the gated recurrent unit (GRU) is used as the recurrent unit, i.e., the RConv(⋅) layer in Eq. (2) updates h.sub.t, given the input x.sub.t")
Regarding claim 5, Metaxas teaches: The scanner of claim 4, wherein the quantized value of the bin is a weight scaling an output of the recurrent unit. ([0059], "Next, Recurrent-MZ outputs (i.e., output images 22) were quantified over all the six (6) permutations of the M=3 input images 20, using the average RMSE (μ.sub.RMSE) and the standard variance of the RMSE (σ.sub.RMSE), calculated with respect to the ground truth image 28 (herein represented by I):")
Regarding claim 6, Metaxas teaches: The scanner of claim 4, wherein the quantized value of the bin is a mask filtering an output of the recurrent unit. ([0059], "Next, Recurrent-MZ outputs (i.e., output images 22) were quantified over all the six (6) permutations of the M=3 input images 20, using the average RMSE (μ.sub.RMSE) and the standard variance of the RMSE (σ.sub.RMSE), calculated with respect to the ground truth image 28 (herein represented by I):")
Regarding claim 7, Metaxas teaches: The scanner of claim 4, wherein the quantized value of the bin is a function of the depth segment modifying an output of the recurrent unit. ([0059], "Next, Recurrent-MZ outputs (i.e., output images 22) were quantified over all the six (6) permutations of the M=3 input images 20, using the average RMSE (μ.sub.RMSE) and the standard variance of the RMSE (σ.sub.RMSE), calculated with respect to the ground truth image 28 (herein represented by I):")
Regarding claim 8, Metaxas teaches: The scanner of claim 1, wherein the scene includes a target object and wherein the rendered one or multiple images include images of one or multiple layers of the target object. ([0031], "The one or more output image(s) 22 is/are digitally propagated to one or more user-defined or automatically generated surface(s) 30 (seen in FIG. 1B). The system 2 includes a computing device 100 that contains one or more processors 102 therein and image processing software 104 that incorporates the trained recurrent neural network 10. The computing device 100 may include, as explained herein, a personal computer, laptop, tablet PC, remote server, application-specific integrated circuit (ASIC), or the like, although other computing devices 100 may be used (e.g., devices that incorporate one or more graphic processing units (GPUs))."; Examiner's Note - Claim language could apply to any image rendered.)
Regarding claim 9, Metaxas teaches: The scanner of claim 8, further comprising: an emitter configured to emit a set of waves in parallel directions of propagation to penetrate a sequence of layers of the target object forming the structure of the target object; and a receiver configured to measure intensities of the set of waves modified by penetration through the layers of the target object. ([0036], "Other types of microscope devices 110 include by way of example: a super-resolution microscope, a STED microscope, a PALM/STORM-based microscope a confocal microscope, a confocal microscope with single photon or multi-photon excited fluorescence, a second harmonic or high harmonic generation fluorescence microscope, a light-sheet microscope, a structured illumination microscope, a TIRF-based microscope, a computational microscope, a ptychographic microscope, an optical coherence tomography (OCT) microscope, or a holographic microscope.")
Regarding claim 11, Metaxas teaches: The scanner of claim 1, wherein the at least one processor is further configured to produce an image of the structure of the scene, based on the rendered one or multiple images. ([0031], "FIG. 1A illustrates one embodiment of a system 2 that uses a trained recurrent neural network 10 that receives or uses a plurality of input images 20 obtained at different depths (e.g., z distances) within a sample 12 to generate one or more output image(s) 22 of a sample 12 (or one or more object(s) in the sample 12). The one or more output image(s) 22 is/are digitally propagated to one or more user-defined or automatically generated surface(s) 30 (seen in FIG. 1B). The system 2 includes a computing device 100 that contains one or more processors 102 therein and image processing software 104 that incorporates the trained recurrent neural network 10.")
Regarding claim 12, Metaxas teaches: The scanner of claim 1, wherein the at least one processor is configured to collect the depth information from a storage device. ([0040], "The GUI 108 may also have a number of pre-defined or arbitrary user-defined or automatically generated surfaces 30 that the user may choose from. These may include planes at different depths, planes at different cross-sections, planes at different tilts, curved or other 3D surfaces that are selected using the GUI 108. This may also include a depth range within the sample 12 (e.g., a volumetric region in the sample 12). The GUI 108 tools may permit the user to easily scan along the depth of the sample 12.")
Regarding claim 13, claim 13 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A method of image reconstruction of a structure of a scene
Regarding claim 14, claim 14 has been analyzed with regard to claim 2 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A method of image reconstruction of a structure of a scene
Regarding claim 15, claim 15 has been analyzed with regard to claim 9 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A method of image reconstruction of a structure of a scene
Regarding claim 16, claim 16 has been analyzed with regard to claim 11 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A method of image reconstruction of a structure of a scene
Regarding claim 17, claim 17 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A non-transitory computer-readable storage medium having stored thereon a program executable by a processor for performing a method for image reconstruction of a structure of a scene, the method comprisin
Regarding claim 18, claim 18 has been analyzed with regard to claim 2 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A non-transitory computer-readable storage medium having stored thereon a program executable by a processor for performing a method for image reconstruction of a structure of a scene, the method comprising
Regarding claim 19, claim 19 has been analyzed with regard to claim 9 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A non-transitory computer-readable storage medium having stored thereon a program executable by a processor for performing a method for image reconstruction of a structure of a scene, the method comprising
Regarding claim 20, claim 20 has been analyzed with regard to claim 11 and is rejected for the same reasons of obviousness as used above as well as in accordance with Metaxas further teaching on: A non-transitory computer-readable storage medium having stored thereon a program executable by a processor for performing a method for image reconstruction of a structure of a scene, the method comprising
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Metaxas et al. (U.S. Patent Publication No. 2026/0080553-A1, hereinafter “Metaxas”) in view of Xue et al. (Xue, W., Nachum, I.B., Pandey, S., Warrington, J., Leung, S., Li, S. (2017). Direct Estimation of Regional Wall Thicknesses via Residual Recurrent Neural Network. In: Niethammer, M., et al. Information Processing in Medical Imaging. IPMI 2017. Lecture Notes in Computer Science(), vol 10265. Springer, Cham. https://doi.org/10.1007/978-3-319-59050-9_40, hereinafter “Xue”).
Regarding claim 2, Metaxas does not teach: The scanner of claim 1, wherein the at least one processor is further configured to:process the measurements with a sparse reconstruction network to recover a sparse structure of the scene along its depth; and quantize the sparse structure into a sequence of bins corresponding to a sequence of depth segments along the depth of the scene, such that each bin includes a quantized value of the sparse structure for a corresponding depth segment of the sequence of depth segments, wherein the sequence of bins has a one-to-one mapping with a sequence of time segments forming the period of time.
However, Xue does teaches: The scanner of claim 1, wherein the at least one processor is further configured to:process the measurements with a sparse reconstruction network to recover a sparse structure of the scene along its depth; and quantize the sparse structure into a sequence of bins corresponding to a sequence of depth segments along the depth of the scene, such that each bin includes a quantized value of the sparse structure for a corresponding depth segment of the sequence of depth segments, (2.2 Overview of the Proposed Method, "As shown in Fig. 2, two paths are included in ResRNN: with the CNN path , each frame in the sequence is independently processed by the CNN network, forming a low dimensional embedding of the cardiac images, from which the RWT is preliminarily estimated with another fully connected layer;"; Examiners Note - Convolutional layers are sparse and the CNN embeddings in prior art maps to bins)
wherein the sequence of bins has a one-to-one mapping with a sequence of time segments forming the period of time. (2.4 Residual Estimation with the RNN Path, "With CNN embedding , we first deploy a temporal RNN with the frame index in a cardiac sequences as time step to predict the values of RWT for each frame f taking account of the dependencies between neighboring frames (See Temporal RNN in Fig. 4).")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify recurrent neural network for volume and motion estimation (Metaxas) to quantize the sparse structure into a sequence of bins (Xue) because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, recurrent neural network for volume and motion estimation as modified by sparse structure into a sequence of bins can yield a predictable result of allowing the recurrent neural network to better correspond the depth data in the recurrent neural network creating more accurate estimations. Thus, a person of ordinary skill would have appreciated including in recurrent neural network for volume and motion estimation the ability to quantize the sparse structure into a sequence of bins since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jinsu Hwang whose telephone number is (703)756-1370. The examiner can normally be reached Mon 6am-8am, 3pm-9pm EST; Thu 12pm - 2pm EST; Fri 12pm - 8pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JINSU HWANG/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights