DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
This office action is in response to application filed on 26 April 2024.
The IDS filed on 13 May 2024 has been considered.
Claims 1-14 and 16-20 are pending in the application.
Limitations appearing inside of {} are intended to indicate the limitations not taught by said prior art(s)/combinations..
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 which recites
a plurality of pipeline logics connected in series in a pipeline (line 4), and
the plurality of the pipeline logics are configured to run in parallel (lines 7-8).
It is unclear how the pipleline logics may run in parallel while they are connected in series. The specification provides support in at least “The plurality of pipeline logics can be configured to run in parallel” (¶[0004]), although the remainder of the disclosure provides support for computing cores configured to be run in parallel, rather than the pipeline logics, in at least (¶[0046]) and Fig 3B. Rational for parallel computing cores is provided in at least (¶[0046]) “Unlike a conventional computing system with a single computing core, the computing system of the present disclosure uses parallel processing, through the plurality of computing cores 352a-352n, to render images. In this way, an amount of data processing (e.g., image rendering) performed by each computing core is reduced, thereby significantly accelerating a rendering speed of the computing system. Furthermore, because the computing system can take advantage of parallel processing, the plurality of computing cores 352a-352n can simultaneously execute computations associated with the pipeline logics. The plurality of image portions 354a-354n+1 can be later combined into the image 354.”. The specification is silent on how and why “pipeline logics are configured to run in parallel”. For the purpose of examination, the claim is interpreted as the computing cores are configured to run in parallel.
Claims 2-14, and 16 are rejected as they depend from a rejected claim.
Claim 16 recites “the plurality of the computing cores” in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim 16 recites “an image” in line 2. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites “an image” in line 1. It is unclear if the limitation, “an image”, of claim 16 this is the same image as claim 1 or if it is a new image. If it is the same image, consider “the [[an]] image” to provide sufficient antecedent basis.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 16 and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by "Ma" (Kwan-Liu. Ma, J. S. Painter, C. D. Hansen and M. F. Krogh, "A data distributed, parallel algorithm for ray-traced volume rendering," Proceedings of 1993 IEEE Parallel Rendering Symposium, San Jose, CA, USA, 1993, pp. 15-22, doi: 10.1109/PRS.1993.586080.).
Regarding claim 1, Ma teaches a computing core for rendering an image comprising: a position encoding logic configured to transform coordinates and directions of a plurality of sampling points corresponding to a portion of the image into high dimensional representations (Ma, [§4.1, p19, col 1, ¶3]; The node program begins by receiving its data-space partitioning information and then its portion of the data (i.e., image portions; see Fig 3, shown below, exhibiting a ray sampling where each sample represents coordinates along the ray, i.e., direction
PNG
media_image1.png
427
461
media_image1.png
Greyscale
)
from the host. It then updates the transfer function and the transform matrices. [§3.1, p 16, col 2, ¶1]; the image sample density is higher than the volume data sample density); and
a plurality of pipeline logics connected in series in a pipeline (Ma, [§ 4, p 18, col 2, ¶ 1]; a data distributor, a renderer, and an image compositor (i.e., pipeline logics); a set of (i.e., a plurality) “node” processes that perform the actual rendering and compositing; Our renderer is a basic renderer and is not highly tuned for best performance),
wherein the plurality of pipeline logics are configured to output, based on the high dimensional representation of the coordinates and the high dimensional representation of the directions, intensity and color values of pixels corresponding to the portion of the image in one pipeline cycle (Ma, [§3.1, p 16, Col 2, ¶1]; At each sample point (i.e., coordinates) on the ray (i.e., direction), a color and an opacity are computed),
wherein the plurality of the pipeline logics are configured to run in parallel a portions of the image (Ma, [§3.4, p18, col 1, ¶1]; accomplish the compositing in parallel: sibling nodes in the tree may be processed concurrently. [§4.1, p18, col1, ¶1]; parallel volume renderer).
Regarding claim 16, Ma teaches the computing core according to claim 1. Ma further teaches wherein the plurality of the computing cores are configured to render a portion of an image in parallel (Ma, [abstract]; volume rendering algorithm and a parallel image compositing method [§3, p16, col 2, ¶1]; divide the data up into smaller subvolumes distributed to multiple computers, render them separately and locally, and combine the resulting images in an incremental fashion).
Regarding claim 17, Ma teaches a computer-implemented image rendering method comprising:
dividing, by a computing system, an image to be rendered into rows of image portions (Ma, [§3.2, p16, col 2, ¶1]; The divide-and-conquer algorithm requires that we partition the input data into subvolumes; and See Fig 1; [§4.1, p19, col1, ¶3]; The node program begins by receiving its data-space partitioning information and then its portion of the data from the host.);
obtaining, by the computing system, for each image portion, coordinates and directions of sampling points corresponding to pixels of the image portion (Ma, [§3.1, p16, col 2, ¶1]; An image is constructed in image order by casting rays from the eye through the image plane and into the volume of data; and See Fig 3 Correct Ray Sampling exhibits coordinates and directions of sample points; [§4.1, p19, col1, ¶3]; It then updates the transfer function and the transform matrices.);
transforming, by the computing system, for each image portion, the coordinates and directions of the sampling points into high dimensional representations (Ma, [§3.2, p16, col 2, ¶1]; the image sample density is higher than the volume data sample density.);
determining, by the computing system, through a computing core, based on the high dimensional representations, intensity and color values of the pixels (Ma, [§3.2, p16, col 2, ¶1]; At each sample point on the ray, a color and an opacity are computed; [§4.1, p19, col1, ¶3]; the nodes all execute their own copy of the renderer.); and
reconstructing, by the computing system, the image based on intensity and color values of pixels of the rows of image portions (Ma, [§3.2, p16, col 2, ¶1]; The host then broadcasts the opacity/colormap and the transformation information to the nodes. Finally, the host performs an 1/0 servicing loop which receives the rendered portions of the image from the nodes; [§4.1, p19, col1, ¶3]; They synchronize after the rendering and before entering the compositing phase. Once the compositing is finished, each node has a portion of the image that they then send back to the host for display.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of “Tancik” (Matthew Tancik and Pratul P. Srinivasan and Ben Mildenhall and Sara Fridovich-Keil and Nithin Raghavan and Utkarsh Singhal and Ravi Ramamoorthi and Jonathan T. Barron and Ren Ng. “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains”. (2020) (available at https://arxiv.org/abs/2006.10739))
Regarding claim 3, Ma teaches the computing core according to claim 1. Ma does not explicitly disclose wherein the position encoding logic is configured to execute Fourier feature mapping to transform the coordinates and the directions of the sampling points to the high dimensional representation of the coordinates and the high dimensional representation of the directions, respectively.
However, Tancik, a similar field of endeavor of using coordinate-based MLPs to represent 3D shapes in computer vision and graphics pipelines, teaches wherein the position encoding logic is configured to execute Fourier feature mapping to transform the coordinates and the directions of the sampling points to the high dimensional representation of the coordinates and the high dimensional representation of the directions, respectively (Tancik, [§4, p4, ¶4]; we use a Fourier feature mapping to featurize input coordinates; The function maps input points
v
2
∈
[
0,1
)
d
to the surface of a higher dimensional hypersphere with a set of sinusoids).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to Fourier feature mapping as taught by Tancik to the invention of Ma. The motivation to do so would be to significantly improve the performance of the imaging task by rendering the higher frequency details of the volume overcoming the spectral bias inherent in coordinate-based MLPs.
Regarding claim 18, Ma teaches the computer-implemented image rendering method according to claim 17. Ma further teaches wherein the coordinates and directions of the sampling points are transformed into the high dimensional representations {based on a Fourier feature mapping technique} (Ma, [§3.2, p16, col 2, ¶1]; At each sample point on the ray, a color and an opacity are computed using trilinear interpolation from the data values at each of the eight nearest voxels.). Ma does not explicitly disclose based on a Fourier feature mapping technique.
However, Tancik teaches wherein the coordinates and directions of the sampling points are transformed into the high dimensional representations based on a Fourier feature mapping technique (Tancik, [§4, p4, ¶4]; we use a Fourier feature mapping to featurize input coordinates; The function maps input points
v
2
∈
[
0,1
)
d
to the surface of a higher dimensional hypersphere with a set of sinusoids).
Claims 6-8, 13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of “Garbin” (Stephan J. Garbin; Marek Kowalski; Matthew Johnson; Jamie Shotton; Julien Valentin. “FastNeRF: High-Fidelity Neural Rendering at 200FPS” Conference Proceedings: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE. 10-17 Oct. 2021.).
Regarding claim 6, Ma teaches the computing core according to claim 1. Ma does not explicitly disclose wherein the plurality of pipeline logics are configured to encode a machine learning model based on a neural network, and wherein each of the plurality of pipeline logics is configured to perform computations associated with particular neural layers of the neural network.
However, Garbin, a similar field of endeavor of volume rendering, teaches wherein the plurality of pipeline logics are configured to encode a machine learning model based on a neural network (Garbin, [§3.1, p 14328, col 2, ¶1]; A Neural Radiance Field (NeRF) [25] captures a volumetric 3D representation of a scene within the weights of a neural network; and Fig 2, shown above exhibits pipeline logics), and
wherein each of the plurality of pipeline logics is configured to perform computations associated with particular neural layers of the neural network (Garbin, [§4, p 14330, col 1, ¶1]; We model Fpos and Fview of FastNeRF using 8 and 4 layer MLPs respectively, with positional encoding applied to inputs).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include pipelines that encode a machine learning model based on a neural network as taught by Garbin to the invention of Ma. The motivation to do so would be to greatly simplify the traditional reconstruction pipelines used in computer vision.
Regarding claim 7, the combination of Ma and Garbin teaches the computing core according to claim 6. Garbin further teaches wherein the neural network is a neural radiance field (Garbin, [§3.1, p 14328, col 2, ¶1]; A Neural Radiance Field (NeRF) [25]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include a neural radiance field as taught by Garbin to the invention of Ma. The motivation to do so would be to employ a method that effectively learns an implicit volumetric representation of the scene that can be rendered from novel viewpoints.
Regarding claim 8, the combination of Ma and Garbin teaches the computing core according to claim 7. Garbin further teaches wherein the neural radiance field is encoded through the neural layers of the neural network (Garbin, [§4, p 14330, col 1, ¶1]; We model Fpos and Fview of FastNeRF using 8 and 4 layer MLPs respectively, with positional encoding applied to inputs) (Garbin, [§4, p 14330, col 1, ¶1]; We model Fpos and Fview of FastNeRF using 8 and 4 layer MLPs respectively, with positional encoding applied to inputs).
Regarding claim 13, Mat teaches the computing core according to claim 1. Ma does not explicitly teach wherein the high dimensional representation of the coordinates comprises 63 dimensions and the high dimensional representation of the directions comprises 27 dimensions.
However, Garbin teaches wherein the high dimensional representation of the coordinates comprises 63 dimensions and the high dimensional representation of the directions comprises 27 dimensions (Garbin, [§3.3], shown below, k samples of space coordinates, and l samples of direction coordinates, and a standard NeRF model with k=l=1024; cashing that is capable of processing up to 1024 samples of space coordinates and direction coordinates may process the dimensions as claimed). Therefore, the cache would be able to process the dimensions as claimed).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include dimensions of coordinates and direction up to 1024 as taught by Garbin to the invention of Ma. The motivation to do so would be to maintain memory requirements to cache outputs and then process the cache at a fraction of the time.
PNG
media_image2.png
658
353
media_image2.png
Greyscale
Regarding claim 19, Ma teaches the computer-implemented image rendering method according to claim 17. Ma further teaches wherein the computing core is configured to execute computations {associated with a machine learning model encoded with a neural radiance field} and the computing core is associated with a row of image portions (Ma, [§3.2, p16, col 2 , ¶1]; The CM-5 is a massively parallel supercomputer; The host program determines which data-space partitioning to use, based on the number of nodes in the CM-5 partition, and sends this information to the nodes.). Ma does not explicitly disclose associated with a machine learning model encoded with a neural radiance field.
However, Garbin teaches wherein the computing core is configured to execute computations associated with a machine learning model encoded with a neural radiance field (Garbin, [§3.1, p 14328, col 2, ¶1]; A Neural Radiance Field (NeRF) [25] captures a volumetric 3D representation of a scene within the weights of a neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include pipelines that encode a machine learning model based on a neural network as taught by Garbin to the invention of Ma. The motivation to do so would be to greatly simplify the traditional reconstruction pipelines used in computer vision.
Regarding claim 20, Ma teaches the computer-implemented image rendering method according to claim 19. Ma does not explicitly disclose the machine learning model is based on a neural network.
However, Garbin teaches (Garbin, [§3.1, p 14328, col 2, ¶1]; A Neural Radiance Field (NeRF) [25] captures a volumetric 3D representation of a scene within the weights of a neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include pipelines that encode a machine learning model based on a neural network as taught by Garbin to the invention of Ma. The motivation to do so would be to greatly simplify the traditional reconstruction pipelines used in computer vision.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of “Hashimoto” (US20040081340A1).
Regarding claim 4, Ma teaches the computing core according to claim 1. Ma does teaches {a first memory and a second memory coupled to the position encoding logic (Ma, [§5.2, p 20, col 1, ¶1]; memories across 32 high performance workstations) {wherein the first memory is configured to store the high dimensional representation of the coordinates and the second memory is configured to store the high dimensional representation of the directions}, wherein the first memory and the second memory are synchronous random access memory modules (Ma, [§4.1, p19, col 1, ¶1];a message passing library for communications and synchronization, which supports either a hostless model or a host/node model). Ma does not explicitly disclose wherein the first memory is configured to store the high dimensional representation of the coordinates and the second memory is configured to store the high dimensional representation of the directions.
However, Hashimoto, a similar field of endeavor of volume rendering, teaches wherein the first memory is configured to store the high dimensional representation of the coordinates and the second memory is configured to store the high dimensional representation of the directions (Hashimoto, [0201] memory controller 321 deciphers the beam position information … and writes data corresponding to the row/column beam address to the first memory 324 through the fourth memory 327 … configured so as to store two sets of ultrasound volume data corresponding to (R, .theta., .psi.)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include storing the high dimension representations of coordinates and directions in separate memories as taught by Hashimoto the invention of Ma. The motivation to do so would be to raise the speed of processing by simultaneously writing and reading.
Regarding claim 5, the combination of Ma and Hashimoto teaches the computing core according to claim 4. Hashimoto further teaches wherein the first memory and the second memory are first-in-first-out memories (Hashimoto, ¶[0201]; FIFO memory 320), and wherein the first memory is configured to store the high dimensional representation of the coordinates and the second memory is configured to store the high dimensional representation of the directions (Hashimoto, [0201] memory controller 321 deciphers the beam position information … and writes data corresponding to the row/column beam address to the first memory 324 through the fourth memory 327 … configured so as to store two sets of ultrasound volume data corresponding to (R, .theta., .psi.)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include FIFP memory as taught by Hashimoto the invention of Ma. The motivation to do so would be to buffer data exchange at the time of writing and reading data.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Garbin and further in view of “Kim” (US 11869132 B2).
Regarding claim 9, the combination of Ma and Garbin teaches the computing core according to claim 6. The combination does not explicitly disclose wherein the neural network comprises ten neural layers.
However, Kim, a similar field of endeavor of rendering an updated graphical representation of an object, teaches wherein the neural network comprises ten neural layers (Kim, [Col 12:64-66]; neural network representations and/or neural network models as described herein include ten-layer).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include a 10 layer network array as taught by Kim to the combined invention of Ma and Garbin. The motivation to do so would be to improve the performance of the network on the complex problem of volume rendering where each layer is optimized for learning complex features.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of “Iyer” (US 20210097641 A).
Regarding claim 14, Ma teaches the computing core according to claim 1. Ma does not explicitly disclose wherein each of the plurality of pipeline logics comprises a multiply-accumulate array
However, Iyer, a similar field of endeavor of real time graphics processing/rendering, teaches wherein each of the plurality of pipeline logics comprises a multiply-accumulate array (Iyer, ¶[0230]; machine learning pipelines 1552 for performing machine learning operations (e.g., mixed-precision fused multiply-accumulate operations (MAC)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include MAC array as taught by Iyer to invention of Ma. The motivation to do so would be to for performing ray tracing operations.
Allowable Subject Matter
Claim 2, and 10-12 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mildenhall et al., (2020) teaches neural radiance fields.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANDHANA PEDAPATI whose telephone number is 571-272-5325. The examiner can normally be reached M-F 8:30am-6pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANDHANA PEDAPATI/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights