Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Amendment
Applicant submitted an amendment on 6/10/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/15/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Claims
Claims 1-9 are pending in this application and have been considered below.
Claims 1-9 are rejected.
Applicant’s Arguments:
In regards to Argument 1, see remarks (page 16), filed 06/10/2026, Applicant/s state/s that “the abstract of the disclosure is objected to because it is 162 words. The Abstract has been amended to be less than 150 words in length. Withdrawal of the objection to the abstract of the disclosure is respectfully requested.”
In regards to Argument 2, see remarks (page 16), filed 06/10/2026, Applicant/s state/s that “the disclosure is objected to for a number of informalities. Paragraphs 3, 7, 21, 25, 26, 28, 29, 38, 40, 41, 42, 44, and 46 have been amended to correct the identified informalities. Withdrawal of the objection to the disclosure is respectfully requested.”
In regards to Argument 3, see remarks (page 16), filed 06/10/2026, Applicant/s state/s that “Figure 1 is objected to because reference number 102 reads "Dynmaic" and should read "Dynamic." Figure 1 has been amended to correct the typographical error. Withdrawal of the objection to Figure 1 is respectfully requested.”
In regards to Argument 4, see remarks (page 16), filed 06/10/2026, Applicant/s state/s that “Claims 5 and 6 are objected to because of informalities. Claim 5, line 5 has been amended to recite "frames." Claim 6, line 2 has been amended to recite "frames." Withdrawal of the objection to claims 5 and 6 is respectfully requested.”
In regards to Argument 5, see remarks (page 17), filed 06/10/2026, Applicant/s state/s that “neither Guo nor Chen, alone or in combination, teaches or suggests a trained deformation encoding neural network coupled to the input and configured to derive at least one parameter characterizing dynamics of the set of consecutive image frames and to generate an interpolated image frame based on the at least one parameter, or generating an interpolated image frame using the trained deformation encoding neural network by deriving at least one parameter characterizing dynamics of the set of consecutive image frames and generating the interpolated image frame based on the at least one parameter.”
In regards to Argument 6, see remarks (pages 18-19), filed 06/10/2026, Applicant/s state/s that “there is no motivation provided in Chen or Guo to incorporate the training data of Chen for input into the trained model of Guo for analysis. The combination of Chen and Guo would merely result in a set of training data for training the motion network and interpolation network of Guo. In addition, as mentioned Guo specifically teaches that the data input to the trained SVIN is a pair of randomly selected time points or volumes. There is no suggestion in Guo of using an alternative, for example, using a set of consecutive image frames, as input for the trained model. Accordingly, claims 1 and 5 are believed to be allowable over the combination of Guo and Chen.”
In regards to Argument 7, see remarks (page 19), filed 06/10/2026, Applicant/s state/s that “Claim 3 depends from amended claim 1 and incorporates all of the limitations of amended claim 1 and is, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 1. Claim 8 depends from amended claim 5 and incorporates all of the limitations of amended claim 5 and is, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 5. Withdrawal of the rejection of claims 1, 3, 5, and 8 under 35 USC 103 is respectfully requested.”
In regards to Argument 8, see remarks (page 19), filed 06/10/2026, Applicant/s state/s that “Claims 2 and 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Chen and further in view of Hsiao et al. (US 2020/0219262). Claim 2 depends from amended claim 1 and incorporates all of the limitations of amended claim 1 and is, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 1. Claims 6 and 7 depends from amended claim 5 and incorporate all of the limitations of amended claim 5 and are, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 5. Withdrawal of the rejection of claims 2 and 6-7 under 35 USC 103 is respectfully requested.”
In regards to Argument 9, see remarks (page 19), filed 06/10/2026, Applicant/s state/s that “Claims 4 and 9 are rejected under 35 USC 103 as being unpatentable over Guo in view of Chen and further in view of Shi et al. ("Video Frame Interpolation Transformer," 2022). Claim 4 has been amended to correspond to the amendments made to independent claim 1 and claim 9 has been amended to correspond to the amendments made to independent claim 5. Claim 4 depends from amended claim 1 and incorporates all of the limitations of amended claim 1 and is, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 1. Claim 9 depends from amended claim 5 and incorporates all of the limitations of amended claim 5 and is, therefore, allowable for, among other reasons, the same reasons as given above with respect to amended claim 5. Withdrawal of the rejection of claims 4 and 9 under 35 USC 103 is respectfully requested.”
Examiner’s Responses:
In response to Argument 1, see remarks (page 16), filed 06/10/2026, regarding applicant’s argument with respect to withdrawal of the abstract objection has been fully considered and is persuasive. The objection to the abstract has been withdrawn.
In response to Argument 2, see remarks (page 16), filed 06/10/2026, regarding applicant’s argument with respect to withdrawal of the objections to the disclosure have been fully considered and are persuasive. The objections to the specification have been withdrawn.
In response to Argument 3, see remarks, (page 16), filed 06/10/2026, regarding applicant’s argument with respect to the drawing objection has been fully considered and is persuasive. The objection to the drawings has been withdrawn.
In response to Argument 4, see remarks, (page 16), filed 06/10/2026, regarding applicant’s argument with respect to the claim objections to claims 5 and 6 have been fully considered and are persuasive. The objections to claims 5 and 6 have been withdrawn.
In response to Argument 5, see remarks, (page 17), filed 06/10/2026, regarding applicant’s argument that neither Guo nor Chen, alone or in combination, teaches or suggests the amended claim language “a trained deformation encoding neural network coupled to the input and configured to derive at least one parameter characterizing dynamics of the set of consecutive image frames and to generate an interpolated image frame based on the at least one parameter, or generating an interpolated image frame using the trained deformation encoding neural network by deriving at least one parameter characterizing dynamics of the set of consecutive image frames and generating the interpolated image frame based on the at least one parameter” has been considered and has been fully persuasive. Therefore, the 35 U.S.C. 103 rejection (Guo in view of Chen) has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo). The details of the rejections are found below.
The Examiner finds that Krebs teaches on the amended claim language “a trained deformation encoding neural network coupled to the input and configured to derive at least one parameter characterizing dynamics of the set of consecutive image frames and to generate an interpolated image frame based on the at least one parameter” and “generating an interpolated image frame using the trained deformation encoding neural network by deriving at least one parameter characterizing dynamics of the set of consecutive image frames and generating the interpolated image frame based on the at least one parameter.”
Specifically, Krebs teaches systems and methods for performing medical imaging analysis tasks (such as image sequence interpolation) using a machine learning based motion model (Abstract). Once the motion model is trained, it may be applied to calculate deformation fields between the images of a sequence, predict the deformation for non-observed motion to generate a complete image sequence from a single image, or generate missing images between images of an image sequence (motion interpolation). The Examiner interprets a deformation field to be “at least one parameter characterizing the dynamics of the set of consecutive image frames.” The deformation field is calculated using the motion model. Further, Krebs teaches the trained motion model may be used in various medical image analysis tasks such as spatio-temporal registration, motion compensation, etc. (Paragraph [0021]). Once the model is trained, the model receives one or more medical images of an anatomical structure, specifically, pairs of the one or more medical images is input into an encoder of the machine learning based motion model (Paragraphs [0031], [0033]). The one or more medical images may be a plurality of images (sequence of images) from any suitable imaging modality (such as MRI, CT, x-ray, ultrasound, etc.) (Paragraph [0032]). Each image pair is encoded by the encoder network into a respective feature vector (Paragraph [0033]). Next, the one or more feature vectors are mapped to one or more motion vectors using the motion model (Paragraph [0034]). Then, one or more deformation fields representing motion of the anatomical structure is determined based on the one or more motion vectors and the one or more medical images (Paragraph [0035]). Finally, a medical imaging task is performed using the one or more deformation fields (Paragraph [0036]). For example, the medical imaging task may be sequence interpolation. Thus, given a sequence of medical images, sequence interpolation is performed to generate a medical image temporally occurring between two images of the sequence. The sequence interpolation is performed by applying deformation fields to an image in the sequence to generate a medical image temporally occurring after the particular image (Paragraph [0039]).
In response to Argument 6, see remarks (pages 18-19), filed 06/10/2026, regarding applicant’s argument that “there is no motivation provided in Chen or Guo to incorporate the training data of Chen for input into the trained model of Guo for analysis. The combination of Chen and Guo would merely result in a set of training data for training the motion network and interpolation network of Guo. In addition, as mentioned Guo specifically teaches that the data input to the trained SVIN is a pair of randomly selected time points or volumes. There is no suggestion in Guo of using an alternative, for example, using a set of consecutive image frames, as input for the trained model. Accordingly, claims 1 and 5 are believed to be allowable over the combination of Guo and Chen” has been considered and has been fully persuasive. Therefore, the 35 U.S.C. 103 rejection (Guo in view of Chen) has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo). The details of the rejections are found below.
The Examiner finds that Krebs teaches the trained motion model may be used in various medical imaging tasks, such as generating missing images between images of an image sequence (motion interpolation) (Paragraph [0021]). Krebs teaches a method (300) for performing a medical imaging analysis task using a machine learning based motion model after it has been trained (Paragraph [0030], Fig. 3). Therefore, the method for performing an image analysis task (such as interpolation) is performed using the trained motion model. In addition, in regards to the input of “consecutive image frames as input for the trained model,” Krebs teaches, given a sequence of medical images, sequence interpolation is performed to generate a medical image temporally occurring between two images of the sequence (Paragraph [0039]). This implies that the two images are occurring sequentially/consecutively/successively arranged in time.
In response to Argument 7, see remarks (page 19), filed 06/10/2026, regarding applicant’s argument regarding withdrawal of the 35 U.S.C. 103 rejection (Guo in view of Chen) of claims 1, 3, 5, and 8 has been fully considered and are persuasive. Therefore, the 35 U.S.C. 103 rejection (Guo in view of Chen) has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claims 1 and 3 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo) and a new ground(s) of rejection is made for Claims 5 and 8 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. WO 2022/104194, hereafter referred to as Chen). The details of the rejections are found below.
In response to Argument 8, see remarks (page 19), filed 06/10/2026, regarding applicant’s argument regarding withdrawal of the 35 U.S.C. 103 rejection (Guo in view of Chen in further view of Hsaio) of claims 2 and 6-7 has been fully considered and is persuasive. Therefore, the 35 U.S.C. 103 rejection (Guo in view of Chen in further view of Hsaio) has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 2 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo) in further view of Hsaio et al. (U.S. Patent Pub. No. 2020/0219262 A1, hereafter referred to as Hsaio) and a new ground(s) of rejection is made for Claims 6 and 7 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. 2022/104194, hereafter referred to as Chen) in further view of Hsaio et al. (U.S. Patent Pub. No. 2020/0219262 A1, hereafter referred to as Hsaio). The details of the rejections are found below.
In response to Argument 9, see remarks (page 19), filed 06/10/2026, regarding applicant’s argument regarding withdrawal of the 35 U.S.C. 103 rejection (Guo in view of Chen in further view of Hsaio) of claims 4 and 9 has been fully considered and is persuasive. Therefore, the 35 U.S.C. 103 rejection (Guo in view of Chen in further view of Hsaio) has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 4 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo) in further view of Shi et al. (NPL “Video Frame Interpolation Transformer,” 2022, hereafter referred to as Shi) and a new ground(s) of rejection is made for Claim 9 under 35 U.S.C. 103 in view of Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. 2022/104194, hereafter referred to as Chen) in further view of Shi et al. (NPL “Video Frame Interpolation Transformer,” 2022, hereafter referred to as Shi). The details of the rejections are found below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 3 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo).
Regarding Claim 1, Wahrenberg teaches a system (Abstract, Fig. 1, Wahrenberg teaches an apparatus.) for increasing a frame rate of a dynamic image (Abstract, Paragraph [0090], Wahrenberg teaches the apparatus obtains upsampled image data that is representative of the subject at a third time. The method performed may provide an appearance of increased frame rate.), the system comprising: an input for receiving (Paragraph [0053], Fig. 1, Wahrenberg teaches the rendering circuitry (24) receives volumetric imaging data from the memory (20).) a set of consecutive image frames of a first dynamic image (Paragraphs [0054], [0008-9], [0138], Fig. 10, Wahrenberg teaches the rendering circuitry (24) renders a first image frame (32) that is representative of the anatomical region at a first time T0, and a second image frame (34) that is representative of the anatomical region at a second time T1. The first image frame (32) and second image frame (34) may be adjacent frames of a sequence of frames. Also, see below, Fig. 10, acquisition of successive frames. The dots (200, 201, 202) represent the acquisition of respective frames. The arrangement of the dots from left to right is representative of the placement of the acquisitions in time.),
PNG
media_image1.png
501
693
media_image1.png
Greyscale
PNG
media_image2.png
100
306
media_image2.png
Greyscale
PNG
media_image3.png
80
806
media_image3.png
Greyscale
wherein the first dynamic image has a first plurality of image frames and a first frame rate (Paragraphs [0138], [0008], [0005], Fig. 10, Wahrenberg teaches acquisition of respective frames (200, 201, 202). Each acquisition is acquired at a single time point, which is represented by a dot. In an example, ultrasound imaging of a mitral valve is performed at 11 frames per second (fps).); (Paragraph [0106], Wahrenberg teaches the interpolation circuitry (28) outputs a set of video data comprising the image data for the original frames and for the interpolated frames. The rendering circuitry displays the resulting animation. The animation comprises the original sequence of frames and the intermediate frames that have been generated. The Examiner interprets the original frames are the “first plurality of image frames” and the interpolated frames are the “one or more interpolated image frames.” The “second plurality of frames” is the video data comprising both the original and interpolated frames.), and to generate a second dynamic image using the second plurality of image frames (Paragraphs [0106], [0003], Wahrenberg teaches outputting a set of video data comprising the image data for the original frames and for the interpolated frames. The resulting animation is displayed. The Examiner interprets video data including the images to be a dynamic image since a moving/dynamic image is effectively a 3D movie/short video (see Wahrenberg, Paragraphs [0003], [0006]).),
the second dynamic image having a second frame rate higher than the first frame rate (Paragraphs [0090], [0112], Wahrenberg teaches a previous frame was acquired at t=10 s and a new frame came in at t=13 s. If a target frame rate is 30 fps (image is reconstructed in 1/30th of a second), the interpolation circuitry (28) generates 90 frames and displays them at 30 fps.).
Wahrenberg does not explicitly disclose a trained deformation encoding neural network coupled to the input and configured to derive at least one parameter characterizing dynamics of the set of consecutive image frames and to generate an interpolated image frame based on the at least one parameter and a post-processing module coupled to the trained deformation encoding neural network and configured to receive one or more interpolated image frames from the deformation encoding neural network.
Krebs is in the same field of art of performing interpolation on a sequence of medical images to generate an image temporally occurring between two images of the plurality of medical images. Further, Krebs teaches a trained deformation encoding neural network (Paragraph [0030], Fig. 3, Krebs teaches once trained, the motion model may be applied to perform medical imaging analysis tasks such as, e.g., image registration, image synthesis, sequence interpolation (e.g., temporal hyper-resolution), and sequence extrapolation (e.g. sequence prediction). Fig. 3 shows a method for performing a medical imaging analysis task using a machine learning based motion model. The Examiner interprets “once trained… the motion model may be applied” to indicate that the motion model is trained prior to being applied/performing image analysis tasks such as the method shown in Fig. 3.) coupled to the input (Paragraph [0033], Krebs teaches inputting pairs of the one or more medical images into an encoder network of the machine learning based motion model.) and configured to derive at least one parameter characterizing dynamics of the set of consecutive image frames (Paragraphs [0033-35], Fig. 1, Krebs teaches determining one or more feature vectors from the one or more medical images by inputting pairs of one or more medical images into an encoder network of the machine learning motion based model. Each image pair is separately encoded by the encoder network into a respective feature vector. The feature vectors represent low dimensional encodings of the motion between the image pairs in the latent space. The feature vectors are then mapped to one or more motion vectors. Then, one or more deformation fields representing motion of the anatomical structure is determined based on the one or more motion vectors and at least one or the one or more medical images.) and to generate an interpolated image frame based on the at least one parameter (Paragraphs [0030], [0036], [0039], Krebs teaches performing a medical imaging task using one or more deformation fields. In one embodiment, the medical imaging analysis task is sequence interpolation. Given a sequence of medical images as the one or more medical images, sequence interpolation is performed to generate a medical image temporally occurring between two images of the sequence. Sequence interpolation is performed by applying the deformation fields to a particular image in the sequence of medical images to generate a generated medical image temporally occurring after that particular image. The generated medical image depicts non-observed motion occurring between the two images of the sequence.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg by using a machine learning motion model to determine deformation fields representing motion of the anatomical structure in the sequence/pair of consecutive images and subsequently perform sequence interpolation by applying the deformation fields to the image that is taught by Krebs, to make the invention that generates a generates medical image temporally occurring after the first image to depict non-observed motion occurring between the two images of the sequence; thus, one of ordinary skilled in the art would be motivated to combine the references since the motion model may estimate the deformation between images more accurately than traditional registration methods since the model is able to extract intrinsic motion parameters that uniquely characterize the underlying organ motion and can be used to model and understand various motion patterns (Krebs, Paragraphs [0046], [0004]). In addition, interpolating between frames may result in smooth playback i.e., the moving image will appear to move smoothly (Wahrenberg, Paragraphs [0008], [0012]).
Wahrenberg in view of Krebs does not explicitly disclose a post-processing module coupled to the trained deformation encoding neural network and configured to receive one or more interpolated image frames from the deformation encoding neural network.
Guo is in the same field of art of performing temporal medical image interpolation for dynamic images. Further, Guo teaches a post-processing module coupled to the trained deformation encoding neural network and configured to receive one or more interpolated image frames from the deformation encoding neural network (Section 3. Proposed Method, Fig. 2, Guo teaches further refining the coarse intermediate images by the volumetric interpolation network by using a regression-based module to constrain the interpolation to follow the patterns of cardiac biological motion.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs by coupling a post-processing module to the motion model that is taught by Guo, to make the invention that refines the interpolated medical images by constraining the intermediate motion field; thus, one of ordinary skilled in the art would be motivated to combine the references to ensure the estimated interpolated frames between the two input frames follow plausible or physically possible patterns of cardiac biological motion (Guo, Section 3. Proposed Method).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 3, Wahrenberg in view of Krebs in further view of Guo discloses the system according to claim 1, wherein the first dynamic image and the second dynamic image are magnetic resonance dynamic images (4.1 Materials and implementation details, Fig. 5, Guo teaches interpolation of images from the ACDC dataset, which contains 4D magnetic resonance (MR) cardiac cine images. The first dynamic image is made up of the input images I0 and I1, which represent images taken at two random time points within the cardiac motion. The “second” dynamic image includes the input images I0 and I1, with the intermediate interpolated images It1/4 and It1/2 being placed “in-between” them.).
Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo) in further view of Hsiao et al. (U.S. Patent Pub. No. 2020/0219262 A1, hereafter referred to as Hsiao).
Regarding Claim 2, Wahrenberg in view of Krebs in further view of Guo discloses the system according to claim 1.
Wahrenberg in view of Krebs in further view of Guo does not explicitly disclose wherein the set of consecutive image frames comprises four consecutive image frames.
Hsiao is in the same field of art of learning the dynamic temporal features of cardiac MRI images using a neural network. Further, Hsiao teaches wherein the set of consecutive image frames comprises four consecutive image frames (Paragraph [0051], Fig. 2, Hsiao teaches a sliding window of four consecutive frames was used as the network input.).
PNG
media_image4.png
430
735
media_image4.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs in further view of Guo by providing four consecutive image frames as input to the neural network/model that is taught by Hsiao, to make the invention that interpolates consecutive MRI frames using an input set of four consecutive medical image frames; thus, one of ordinary skilled in the art would be motivated to combine the references because by simultaneously providing a set of four consecutive image frames to the neural network/model, it provides the network with temporal context of each frame to address the temporal relationship between image frames (Hsiao, Paragraphs [0049] and [0052]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Guo et al. (NPL “A Spatiotemporal Volumetric Interpolation Network for 4D Dynamic Medical Image,” 2020, hereafter referred to as Guo) in further view of Shi et al. (NPL “Video Frame Interpolation Transformer,” 2022, hereafter referred to as Shi).
In regards to Claim 4, Wahrenberg in view of Krebs in further view of Guo discloses the system according to claim 1.
Wahrenberg in view of Krebs in further view of Guo does not explicitly disclose wherein the trained deformation encoding neural network comprises a transformer-based deep learning architecture.
Shi is in the same field of art of interpolating image frames to temporally up-sample an input video by synthesizing new frames between existing ones. Further, Shi teaches wherein the trained deformation encoding neural network comprises a transformer-based deep learning architecture (1 Introduction, 3.1 Learning Deep Features, Shi teaches the Video Frame Interpolation Transformer (VFIT) for video interpolation. A Transformer-based encoder-decoder architecture is used to extract the deep hierarchical features from images.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs in further view of Guo by applying a transformer-based architecture to extract hierarchical feature representations to capture the multi-scale motion information from images that is taught by Shi, to make the invention that interpolates image frames using the derived motion information to increase the frame rate of the input image to generate a second dynamic image with a higher frame rate; thus, one of ordinary skilled in the art would be motivated to combine the references to overcome the drawbacks associated with CNN-based architectures such as their inefficiency in exploiting long-range information and are thus less efficient in synthesizing high-quality image/video frames (Shi, 1. Introduction). Therefore, transformer-based models are better suited for the task of video interpolation.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claims 5 and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. WO 2022/104194, hereafter referred to as Chen).
Regarding Claim 5, Wahrenberg teaches a method for increasing a frame rate of a dynamic image (Paragraphs [0002], [0090], Wahrenberg teaches a method for image processing, for example for temporal upsampling of frames. The method performed may provide an appearance of increased frame rate.), the method comprising: receiving a first dynamic image having a first plurality of image frames and a first frame rate (Paragraphs [0138], [0008], [0005], Fig. 10, Wahrenberg teaches the acquisition of respective frames (200, 201, 202). Each acquisition is acquired at a single time point, which is represented by a dot. In an example, ultrasound imaging of a mitral valve is performed at 11 frames per second (fps).); (Paragraph [0106], Wahrenberg teaches the interpolation circuitry (28) outputs a set of video data comprising the image data for the original frames and for the interpolated frames. The rendering circuitry displays the resulting animation. The animation comprises the original sequence of frames and the intermediate frames that have been generated. The Examiner interprets the original frames are the “first plurality of image frames” and the interpolated frames are the “one or more interpolated image frames.”); and generating a second dynamic image using the second plurality of image frames (Paragraphs [0106], [0003], Wahrenberg teaches outputting a set of video data comprising the image data for the original frames and for the interpolated frames. The resulting animation is displayed. The Examiner interprets video data including the images to be a dynamic image since a moving/dynamic image is effectively a 3D movie/short video (see Wahrenderg, Paragraphs [0003], [0006]).), the second dynamic image having a second frame rate higher than the first frame rate (Paragraphs [0090], [0112], Wahrenberg teaches a previous frame was acquired at t=10 s and a new frame came in at t=13 s. If a target frame rate is 30 fps (image is reconstructed in 1/30th of a second), the interpolation circuitry (28) generates 90 frames and displays them at 30 fps.).
Wahrenberg does not explicitly disclose selecting a plurality of sets of consecutive image frames from the first plurality of image frames of the first dynamic image; for each set of consecutive image frames: providing the set of consecutive image frames to a trained deformation encoding neural network; generating an interpolated image frame using the trained deformation encoding neural network by deriving at least one parameter characterizing dynamics of the set of consecutive image frames and generating the interpolated image frame based on the at least one parameter; and storing the interpolated image frame in data storage.
Krebs is in the same field of art of performing interpolation on a sequence of medical images to generate an image temporally occurring between two images of the plurality of medical images. Further, Krebs teaches(Paragraphs [0030-32], Fig. 3, Krebs teaches a method 300 for performing a medical imaging analysis task using a machine learning motion model. At step 302, one or more medical images of an anatomical structure is/are received. The one or more images may be a plurality of images, such as a sequence of medical images, such as MRI images. The Examiner interprets the machine learning motion model is trained since paragraph [0030] states “once trained, the motion model may be applied… to perform medical imaging analysis tasks…Fig. 3 shows a method 300 for performing a medical imaging analysis task using a machine learning based motion model,” which implies the model is trained prior to performing the image analysis task (method 300).);
PNG
media_image5.png
729
397
media_image5.png
Greyscale
generating an interpolated image frame (Paragraphs [0030], [0036], [0039], Krebs teaches performing a medical imaging task using one or more deformation fields. In one embodiment, the medical imaging analysis task is sequence interpolation. Given a sequence of medical images as the one or more medical images, sequence interpolation is performed to generate a medical image temporally occurring between two images of the sequence. Sequence interpolation is performed by applying the deformation fields to a particular image in the sequence of medical images to generate a generated medical image temporally occurring after that particular image. The generated medical image depicts non-observed motion occurring between the two images of the sequence.) using the trained deformation encoding neural network by deriving at least one parameter characterizing dynamics of the set of consecutive image frames (Paragraphs [0030], [0032], [0035], Fig. 3 (reference character 308), Krebs teaches determining one or more deformation fields representing motion of the anatomical structure based on the one or more motion vectors and at least one or the one or more medical images. The one or more medical images may be a plurality of images such as a sequence of medical images. The Examiner interprets the machine learning motion model is trained since paragraph [0030] states “once trained, the motion model may be applied… to perform medical imaging analysis tasks…Fig. 3 shows a method 300 for performing a medical imaging analysis task using a machine learning based motion model,” which implies the model is trained prior to performing the image analysis task (method 300).) and generating the interpolated image frame based on the at least one parameter (Paragraphs [0030], [0036], [0039], Krebs teaches performing a medical imaging task using one or more deformation fields. In one embodiment, the medical imaging analysis task is sequence interpolation. Given a sequence of medical images as the one or more medical images, sequence interpolation is performed to generate a medical image temporally occurring between two images of the sequence. Sequence interpolation is performed by applying the deformation fields to a particular image in the sequence of medical images to generate a generated medical image temporally occurring after that particular image. The generated medical image depicts non-observed motion occurring between the two images of the sequence.); and storing the interpolated image frame in data storage (Paragraphs [0041], [0039], Krebs teaches storing the results of the medical image analysis task on a memory or storage of a computer system. The medical imaging analysis task may be sequence interpolation to create a medical image temporally occurring between two images of the sequence.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg by using a machine learning motion model to determine deformation fields representing motion of the anatomical structure in the sequence of consecutive images and subsequently perform sequence interpolation by applying the deformation fields to the image that is taught by Krebs, to make the invention that generates a generates medical image temporally occurring after the first image to depict non-observed motion occurring between the two images of the sequence; thus, one of ordinary skilled in the art would be motivated to combine the references since the motion model may estimate the deformation between images more accurately than traditional registration methods since the model is able to extract intrinsic motion parameters that uniquely characterize the underlying organ motion and can be used to model and understand various motion patterns (Krebs, Paragraphs [0046], [0004]). In addition, interpolating between frames may result in smooth playback i.e., the moving image will appear to move smoothly (Wahrenberg, Paragraphs [0008], [0012]).
Wahrenberg in view of Krebs does not explicitly disclose selecting a plurality of sets of consecutive image frames from the first plurality of image frames of the first dynamic image.
Chen is in the same field of art of modifying medical image data by altering an amount of time between any two frames of captured medical image data, with the modified image data having an interpolated image frame created from other frames in the captured medical image data. Further, Chen teaches selecting a plurality of sets of consecutive image frames from the first plurality of image frames of the first dynamic image (Paragraph [0009], Chen teaches splitting the captured image data into a plurality of sub-sequences of pre-determined length.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs by selecting multiple sets of consecutive image frames that is taught by Chen, to make the invention that interpolates intermediate image frames between pairs of images for multiple sets; thus, one of ordinary skilled in the art would be motivated to combine the references since a viewer may want to view at multiple time points throughout the imaging duration (Krebs, Paragraph [0006]) and therefore, interpolating multiple sets of images from the sequence of images may enable the viewer to closely view a specific time point/occurrence during the imaging.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 8, Wahrenberg in view of Krebs in further view of Chen discloses the method according to claim 5, wherein the first dynamic image and the second dynamic image are magnetic resonance dynamic images (Paragraph [0045], Wahrenberg teaches obtaining volumetric imaging data that is representative of an anatomical region of a patient or other subject. The imaging data may be obtained using MRI (magnetic resonance imaging).).
Claims 6 and 7 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. 2022/104194, hereafter referred to as Chen) in further view of Hsiao et al. (U.S. Patent Pub. No. 2020/0219262 A1, hereafter referred to as Hsiao).
Regarding Claim 6, Wahrenberg in view of Krebs in further view of Chen discloses the method according to claim 5.
Wahrenberg in view of Krebs in further view of Chen does not explicitly disclose wherein selecting a plurality of sets of consecutive image frames from the first plurality of image frames of the first dynamic image comprises using a sliding window technique.
Hsiao is in the same field of art of learning the dynamic temporal features of cardiac MRI images using a neural network. Further, Hsiao teaches wherein selecting a plurality of sets of consecutive image frames from the first plurality of image frames of the first dynamic image comprises using a sliding window technique (Paragraphs [0049], [0051], Fig. 2, Hsiao teaches a sliding window of four consecutive frames was used as the network input.).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs in further view of Guo by applying a sliding window technique to the series of consecutive image frames to generate various interpolated image frames; thus one of ordinary skill in the art would have been motivated to combine the references to automate the process of generating different frame windows to improve reliability and repeatability of the image set selection process (Hsiao, Paragraph [0012]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 7, Wahrenberg in view of Krebs in further view of Chen discloses the method according to claim 5.
Wahrenberg in view of Krebs in further view of Chen does not explicitly disclose wherein each set of consecutive image frames comprises four consecutive image frames.
Hsiao is in the same field of art of learning the dynamic temporal features of cardiac MRI images using a neural network. Further, Hsiao teaches wherein each set of consecutive image frames comprises four consecutive image frames (Paragraph [0051], Fig. 2, Hsiao teaches a sliding window of four consecutive frames was used as the network input.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs in further view of Guo by providing four consecutive image frames as input to the neural network that is taught by Hsiao, to make the invention that interpolates consecutive MRI frames using an input set of four consecutive medical image frames; thus, one of ordinary skilled in the art would be motivated to combine the references because by simultaneously providing a set of four consecutive image frames to the neural network, it provides the network with temporal context of each frame to address the temporal relationship between image frames (Hsiao, Paragraphs [0049] and [0052]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wahrenberg et al. (U.S. Patent Pub. No. 2021/0049734 A1, hereafter referred to as Wahrenberg) in view of Krebs et al. (U.S. Patent Pub. No. 2020/0311940 A1, hereafter referred to as Krebs) in further view of Chen et al. (Int. Patent. Pub. No. 2022/104194, hereafter referred to as Chen) in further view of Shi et al. (NPL “Video Frame Interpolation Transformer,” 2022, hereafter referred to as Shi).
Regarding Claim 9, Wahrenberg in view of Krebs in further view of Chen discloses the method according to claim 5.
Wahrenberg in view of Krebs in further view of Chen does not explicitly disclose wherein the trained deformation encoding neural network comprises a transformer-based deep learning architecture.
Shi is in the same field of art of interpolating image frames to temporally up-sample an input video by synthesizing new frames between existing ones. Further, Shi teaches wherein the trained deformation encoding neural network comprises a transformer-based deep learning architecture (1 Introduction, 3.1 Learning Deep Features, Shi teaches the Video Frame Interpolation Transformer (VFIT) for video interpolation. A Transformer-based encoder-decoder architecture is used to extract the deep hierarchical features from images.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wahrenberg in view of Krebs in further view of Guo by applying a transformer-based architecture to extract hierarchical feature representations to capture the multi-scale motion information from images that is taught by Shi, to make the invention that interpolates image frames using the derived motion information to increase the frame rate of the input image to generate a second dynamic image with a higher frame rate; thus, one of ordinary skilled in the art would be motivated to combine the references to overcome the drawbacks associated with CNN-based architectures such as their inefficiency in exploiting long-range information and are thus less efficient in synthesizing high-quality image/video frames (Shi, 1. Introduction). Therefore, transformer-based models are better suited for the task of video interpolation.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Krebs et al. (U.S. Patent Pub. No. 2020/0090345 A1) teaches a method and system for motion estimation and modeling in a medical image sequence of a patient. A plurality of frames of the medical image sequence are input into a trained deep neural network. Diffeomorphic deformation fields representing estimated motion between the frames of the medical image sequence input to the trained deep neural network are generated. Future motion, or motion between frames, is predicted from the medical image sequence and at least one predicted next frame is generated using the trained deep neural network.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7120. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights