DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-16 are pending. No claim is amended, cancelled or newly added.
Response to Arguments
Applicant's arguments filed 1/30/2026 have been fully considered but they are not persuasive.
Citing that Fibrosis Metric processor 160 does not include a neural processor there, applicant argues,
Ruggiero only discloses that the labelled voxel data, which includes a binary "fibrosis" label for each data element and a label indicating portions of the lung, is processed by a fibrosis metric processor to compute a fibrosis volume metric from the labelled voxel data and/or that a two-dimensional slice through the labelled voxel data is taken and provided as an overlay to initial lung imaging data. It is evident from these excerpts in Ruggiero that Ruggiero does not teach or suggest using a feed-forward neural network to process labeled voxels describing a scene in 3D, let alone that such processing of labeled voxels using the feed-forward neural network generates a 3D representation of the scene.
In response, the Examiner respectfully points out that because applicant has the opportunity to amend the claims during prosecution, giving a claim its broadest reasonable interpretation (BRI) will reduce the possibility that the claim, once issued, will be interpreted more broadly than is justified [ In re Yamamoto, 740 F.2d 1569, 1571 (Fed. Cir. 1984); In re Zletz, 893 F.2d 319, 321 (Fed. Cir. 1989). (“During patent examination the pending claims must be interpreted as broadly as their terms reasonably allow.”); < In re Prater, 415 F.2d 1393, 1404-05, 162 USPQ 541, 550-51 (CCPA 1969)].
Specifically, the arrangement of claim limitations in claim 1 (and also likewise similar claims 15-16) recited in such a manner that the limitation of “using a feed-forward neural network” falls under interpretive context. Since the limitation is recited after reciting, “processing an input that includes a plurality of labeled voxels describing a scene in three-dimensions (3D)” – Usage if feed-forward neural network can either go hand in hand with antecedent “processing the input” OR “voxels describing a scene in three-dimensions (3D)”. Applicant’s argument relies that “processing the input” is coupled with “using a feed-forward neural network”.
However, under the provisions of BRI, “voxels describing a scene in three-dimensions (3D)” can also be done “using a feed-forward neural network” – which is also a valid interpretation.
Due to the placement of the limitation “using a feed-forward neural network” after two potential antecedents, Examiner contends that reading the limitation along with “voxels describing a scene in three-dimensions (3D)” is equally valid under BRI. And indeed, since Labelled voxel data is generated as output of block 152, prior neural network blocks at units 120 and 140 shapes or preprocesses the eventual labelled voxel data, output at block 152, and thus meeting the limitation.
About Applicant’s arguments of potential lack of Feed-Forward neural network, Examiner wants to point out that neural network of fig. 3A that represents block 140 of fig. 1 is indeed a feed forward neural network, since data flow indicated by arrows show a forward-facing direction.
Examiner’s Note: Examiner encourages reciting limitation in an unambiguous way, so that Examiner does not have any interpretive leeway, to interpret limitation under BRI. For Example,
If the limitation -
processing an input that includes a plurality of labeled voxels describing a scene in three-dimensions (3D), using a feed-forward neural network, to generate a 3D representation of the scene
is recited in the following manner,
processing an input using a feed-forward neural network to generate a 3D representation of the scene, wherein the input data includes a plurality of labeled voxels describing a scene in three-dimensions (3D)
no interpretive leeway is available to the Examiner, but only one singular unambiguous way.
Thus, the rejection of the previous Office Action is maintained herein.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-8, 12-16 is/are rejected under 35 U.S.C. 102 (a)(1) and/or 102 (a)(2) as being anticipated by Ruggiero et al. (US 20250157664 A1, hereinafter Ruggiero).
Regarding claim 1, Ruggiero discloses a method (abstract, ¶1, ¶0018-0022, figs. 1, 4-5, claims 11, 14 and dependents), comprising:
at a device (device 600, fig. 6, ¶0110):
processing an input that includes a plurality of labeled voxels describing a scene in three-dimensions (3D), using a feed-forward neural network, to generate a 3D representation of the scene (A “neural network architecture” may comprise a particular arrangement of one or more neural network layers of one or more neural network types. Neural network types include convolutional neural networks, recurrent neural networks, and feed-forward neural networks, ¶0071, fig. 3.
This block 304 receives the 3D imaging data 302 and applies a three-dimensional convolution neural network operation. The 3D imaging data 302 is received as a tensor … - ¶0079.
In certain variations, the initial lung imaging data comprises three-dimensional computed tomography (CT) image data, ¶0091
In this case, the fibrosis segmentation labelling may form part of the fibrosis segmentation data 142 in FIG. 1, or be based on the output labelled voxel data 152 as generated by the fibrosis model post-processor 150. In FIGS. 2A and 2B, the fibrosis segmentation is binary (i.e., “fibrosis” or “not fibrosis”) and is applied to individual data elements (which may correspond to pixels or pixel areas in the lung imaging data). Two-dimensional slices of fibrosis segmentation labelling may be generated by taking slices through output labelled voxel data 152 along the same planes as used for the axial and coronal slices, ¶0068.
Unit 152 in fig. 1); and
generating a two-dimensional (2D) image of the scene from a given viewpoint, using the 3D representation of the scene (In a case where the lung segmentation model 120 and the lung fibrosis model 140 operate on a two-dimensional input (e.g., on two-dimensional CT images) then the components 120 to 140 may be applied iteratively (e.g., to a plurality of said images) to provide the lung segmentation data 122 and the fibrosis segmentation data 142 as a plurality of two-dimensional data “slices” for different planes through the volume of the lungs, ¶0063
Two-dimensional slices of fibrosis segmentation labelling may be generated by taking slices through output labelled voxel data 152 along the same planes as used for the axial and coronal slices., ¶0068.
For example, if CT imaging data is available in the form of a three-dimensional annotated model and the neural network models are configured to receive two-dimensional inputs, then additional two dimensional slices may be generated from the three-dimensional model in additional to available two-dimensional images with annotations, ¶0101).
Regarding claim 2, Ruggiero discloses the method of claim 1, wherein the input description is manually provided by a user (The lung fibrosis metric 106 may be comparable to a fibrosis volume metric that is calculated based on a set of manual radiologist fibrosis annotations. , ¶0040.
In the present examples, at least some of the “ground truth” labels may be labels as previously manually annotated by radiologists, ¶0075).
Regarding claim 3, Ruggiero discloses the method of claim 1, wherein each of the labeled voxels has a semantic meaning (There can thus be considerable variation between radiologists in their interpretation of the same dataset (or even between annotations of the same radiologist at different points in time). Different experience levels can also make a difference to the annotation of fibrosis and progression determinations., ¶0040
For example, the fibrosis model post-processor 150 may use the lung segmentation data 122 to set a multi-class label within the labelled voxel data 152 (e.g., as a set of multi-channel one-hot values and/or integer labels mapped by a dictionary to the different anatomic portions)., ¶0066. Also see ¶0080).
Regarding claim 4, Ruggiero discloses the method of claim 1, wherein each of the labeled voxels is a voxel labeled with a descriptor of an object represented by the voxel (For example, the fibrosis model post-processor 150 may use the lung segmentation data 122 to set a multi-class label within the labelled voxel data 152 (e.g., as a set of multi-channel one-hot values and/or integer labels mapped by a dictionary to the different anatomic portions)., ¶0066. Also see ¶0080).
Regarding claim 5, Ruggiero discloses the method of claim 1, wherein the feed-forward neural network further processes an input style code to generate the 3D representation of the scene (The left layer 1 processing block 306 receives input data of size [16, 32, 32, 32], applies a three-dimensional convolution neural network operation, and outputs a tensor of size [32, 16, 16, 16]—i.e. as a set of voxels at a lower resolution each having a feature vector of size 32. The output of the left layer 1 processing block 306 is provided to a left layer 2 processing block 308 and also passed across to the right side of the architecture via connection 334. The left layer 2 processing block 308 receives input data of size [32, 16, 16, 16], applies a three-dimensional convolution neural network operation, and outputs a tensor of size [64, 8, 8, 8]—i.e. as a set of voxels at a lower resolution each having a feature vector of size 64, ¶0079. Also see ¶0080).
Regarding claim 6, Ruggiero discloses the method of claim 1, wherein the 3D representation of the scene is a 3D feature map (In this case, a first dimension of the feature vector indicates a probability that a voxel forms part of a background and a second dimension of the feature vector indicates a probability that a voxel exhibits fibrosis. These two probability values for each voxel are then converted into a single segmentation map., ¶0080).
Regarding claim 7, Ruggiero discloses the method of claim 1, wherein the 3D representation of the scene is a voxel grid with features (At the bottom of the architecture is a lower portion 310 comprising a layer 3 processing block 312. The output of the left layer 2 processing block 308 is also passed across to the right side of the architecture via connection 336. The layer 3 processing block 312 receives input data of size [64, 8, 8, 8], applies a three-dimensional convolution neural network operation, and outputs a tensor of size [128, 8, 8, 8]—i.e. as a set of voxels at the same resolution each having a feature vector of size 128. At the output of the lower portion 310, the output of the layer 3 processing block 312 is concatenated with the data passed via connection 336 so as to provide a tensor of size [192, 8, 8, 8]. This tensor is input to a right layer 2 processing block 314, which applies a three-dimensional convolution neural network operation, and outputs a tensor of size [32, 16, 16, 16]—i.e. as a set of voxels at a higher resolution each having a feature vector of size 32. The tensor output from the right layer 2 processing block 314 is then concatenated with the data received along connection 334 so as to provide a tensor of size [64, 16, 16, 16]. This tensor is input to a right layer 1 processing block 316, which applies a three-dimensional convolution neural network operation, and outputs a tensor of size [16, 32, 32, 32]—i.e. as a set of voxels at a higher resolution each having a feature vector of size 16. The tensor output from the right layer 1 processing block 316 is then concatenated with the data received along connection 332 so as to provide a tensor of size [32, 32, 32, 32]. This tensor is then input to a right layer 0 processing block 318, which applies a three-dimensional convolution neural network operation, and outputs a tensor of size [2, 64, 64, 64]—i.e. as a set of voxels at a higher resolution each having a feature vector of size 2. This is then provided as the three-dimensional fibrosis segmentation data 320. In this case, a first dimension of the feature vector indicates a probability that a voxel forms part of a background and a second dimension of the feature vector indicates a probability that a voxel exhibits fibrosis. These two probability values for each voxel are then converted into a single segmentation map., ¶0080).
Regarding claim 8, Ruggiero discloses the method of claim 1, wherein the 3D representation of the scene is a tri-plane representation (sliced 2d representations of the 3d model are shown in axial and coronal plane, see figs. 2a-b, ¶0031, 0068. Therefore, it is understood that the 3d model has another plane, which intersects both axial and coronal plane, also potentially known in medical field as sagittal plane).
Regarding claim 12, Ruggiero discloses the method of claim 1, further comprising, at the device: optimizing the 2D image of the scene (The convolutional neural network architectures described herein may use kernels and/or filters to extract features in two or (preferably) three dimensions., ¶0101).
Regarding claim 13, Ruggiero discloses the method of claim 12, wherein the 2D image of the scene is optimized by a second feed-forward neural network (The convolutional neural network architectures described herein may use kernels and/or filters to extract features in two or (preferably) three dimensions., ¶0101. ).
Regarding claim 14, Ruggiero discloses the method of claim 1, wherein the feed-forward neural network generates the 3D representation of the scene from the input in a single feed-forward step (For example, if CT imaging data is available in the form of a three-dimensional annotated model and the neural network models are configured to receive two-dimensional inputs, then additional two dimensional slices may be generated from the three-dimensional model in additional to available two-dimensional images with annotations. Similarly, in the case that the neural network models are configured to receive three-dimensional inputs and only sets of two-dimensional annotated images are available, machine learning approaches (e.g., based on three-dimensional interpolation or sample selection) may be used to generate three-dimensional annotated data samples from the original two-dimensional annotated images. In certain cases, if physical images with manual hand annotations are available these may be digitalised to generate digital copies for use in training., ¶0101).
Regarding claim 15, Ruggiero discloses a system (abstract, ¶0016, figs. 1, 6, system 600), comprising:
a non-transitory memory storage comprising instructions (608, fig. 6, ¶0110); and
one or more processors (602, fig. 6) in communication with the memory, wherein the one or more processors execute the instructions to (¶0110, fig. 6):
process an input that includes a plurality of labeled voxels describing a scene in three-dimensions (3D), using a feed-forward neural network, to generate a 3D representation of the scene; and generate a two-dimensional (2D) image of the scene from a given viewpoint, using the 3D representation of the scene (see substantively similar claim 1 rejection above).
Regarding claim 16, Ruggiero discloses a non-transitory computer-readable media (memory 608, fig. 6) storing computer instructions which when executed by one or more processors of a device cause the device to (fig. 6, ¶0110):
process an input that includes a plurality of labeled voxels describing a scene in three-dimensions (3D), using a feed-forward neural network, to generate a 3D representation of the scene; and generate a two-dimensional (2D) image of the scene from a given viewpoint, using the 3D representation of the scene (see substantively similar claim 1 rejection above).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ruggiero in view of Deng et al. (US 20240320909 A1, hereinafter Deng).
Regarding claim 9, Ruggiero discloses the method of claim 1, except, wherein the given viewpoint is defined based on an input camera pose.
However, Deng discloses method of generating 3D model from plurality of labelled images, wherein, once trained, machine-learning model 202 may be capable of generating new semantically-labelled two-dimensional images of the object represented in images 204 and segmentation masks 206 from camera positions and viewing angles not represented in images 204 and segmentation masks 206 (¶0048).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Ruggiero with the teaching of Deng of generating a 2D image slice from the 3D model from the viewpoint of the camera pose, because, combining prior art elements according to known method to yield predictable results is obvious. Furthermore, such combination would enhance the versatility of the overall system.
Regarding claim 10, Ruggiero discloses the method of claim 1, except, wherein the given viewpoint is controllable such that different 2D images of the scene are renderable from different given viewpoints, using the 3D representation of the scene.
However, Deng discloses method of generating 3D model from plurality of labelled images, wherein, once trained, machine-learning model 202 may be capable of generating new semantically-labelled two-dimensional images of the object represented in images 204 and segmentation masks 206 from camera positions and viewing angles not represented in images 204 and segmentation masks 206 (¶0048).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Ruggiero with the teaching of Deng of generating a 2D image slice from the 3D model from any arbitrary viewpoint of a camera pose defined by the user to obtain, wherein the given viewpoint is controllable such that different 2D images of the scene are renderable from different given viewpoints, using the 3D representation of the scene, because, combining prior art elements according to known method to yield predictable results is obvious. Furthermore, such combination would enhance the versatility of the overall system.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ruggiero in view of Hao et al. (US 20220180602 A1, hereinafter Hao).
Regarding claim 11, Ruggiero discloses the method of claim 1, except, wherein the 2D image is generated by projecting the 3D representation of the scene to a 2D feature map via a neural radiance field rendering.
However, Hao discloses, method of image generation using neural networks based on one or more semantic features (abstract), wherein, 2D image is generated by projecting the 3D representation of the scene to a 2D feature map via a neural radiance field rendering (¶0050, 0062-0063).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Ruggiero, using the teaching of Hao to generate 2D image by projecting the 3D representation of the scene to a 2D feature map via a neural radiance field rendering, because, combining prior art elements according to known method to yield predictable results is obvious. Furthermore, such combination would enhance the versatility of the overall system.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NURUN FLORA whose telephone number is (571)272-5742. The examiner can normally be reached M-F 9:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571) 272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NURUN FLORA/Primary Examiner, Art Unit 2619