DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-(d) prior to declaration of an interference, a certified English translation of the foreign application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e).
Failure to provide a certified translation may result in no benefit being accorded for the non-English application.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a visual perception core, a hybrid neural engine, a dynamic neural network allocation core in claims 1-10; a temporal familiarity unit, a spatial attention unit in claims 2-10; a 3D transformer in claims 3-5; a sample coordinate generator and a task controller in claims 6-7; top-down attention logic in claim 7; a 1D Ne, a 2D NE and a modulo-based positional encoding unit in claims 8-9; and an output zero pattern extractor, a bit mask generator, a selection information generator, an offloading controller, and a clock controller in claim 10.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 20 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 20, on lines 8-9, the claim ends with the limitation “(Equation)”. However, no equation is recited in the claim, and therefore, the claim is considered indefinite for failing to particularly point out and distinctly claim the subject matter. For prior art rejection, the equation as recited in claim 9 will be considered as the missing equation in this claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 and 11 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Han et al. (“MetaVRain: A 133mW Real-Time Hyper-Realistic 3D-NeRF Processor with 1D-2D Hybrid-Neural Engines for Metaverse on Mobile Devices”, hereinafter Han).
The applied reference has common inventors and/or a common applicant/assignee with the instant application. Based on the earlier publication date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(1).
Applicant may rely on the exception under 35 U.S.C. 102(b)(1)(A) to overcome this rejection under 35 U.S.C. 102(a)(1) by a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application, and is therefore not prior art under 35 U.S.C. 102(a)(1). Alternatively, applicant may rely on the exception under 35 U.S.C. 102(b)(1)(B) by providing evidence of a prior public disclosure via an affidavit or declaration under 37 CFR 1.130(b).
Regarding claim 1, Han teaches an artificial intelligence (AI)-based high-speed and low-power three-dimensional (3D) rendering accelerator (page 50 col. 2 paragraph 5: MetaVRain is fabricated in 28nm CMOS technology, integrating 5K FP8-FP16 configurable MACs with 2MB of SRAM. Its power and throughput can be varied dynamically by selecting operating modes with SW programming, and there are two typical operating modes: power-efficient and high-speed. In power-efficient mode, the chip consumes 133mW while maintaining >30fps. In high-speed mode, it can achieve a maximum of 118fps with 899mW power consumption. In summary, we present an efficient NeRF architecture, BuFF, and we design and fabricate MetaVRain with the proposed BuFF architecture for low-power and real-time 3D rendering) based on a deep neural network (DNN) trained using a weight of the DNN using a plurality of two-dimensional (2D) photos obtained by imaging the same object from several directions and then configured to perform 3D rendering using the same (Figure 2.7.1 “Overview of NeRF w/3 Main Processes” shows the process using a plurality of images of an object as seen from different VR head positions (different viewing directions) and shows that the volumetric rendering is performed using weighted-sum of the DNN INF; page 50 col. 1 paragraph 1: A neural radiance field (NeRF) [1] uses a deep neural network (DNN) to create 3D models by training the DNN to memorize 3D scene geometry from a few photos), the 3D rendering accelerator comprising:
a visual perception core (VPC) (this element is interpreted as hardware under 35 USC 112(f); Figure 2.7.2 shows VPC (Visual Perception Core); page 50 col. 1 paragraph 4: MetaVRain has 3 key hardware blocks: 1) a Visual Perception Core (VPC) to realize the BuFF-based computational cost reduction) configured to create an image plane for an object that is a 3D rendering target from a position and a direction of an observer (Figure 2.7.1 shows an image plane of the viewed object is formed by the position and direction of the VR user), divide the image plane into a plurality of tile units (it is inherent that the image as displayed in an image plane is divided into a plurality of tile units for Tile-wise 3D Transform and Tile-wise polarized sparsity; Figure 2.7.3 shows Tile-wise 3D Transform of an image represented by a plurality of tiles; Figure 2.7.4 shows Tile-wise polarized sparsity where an image is represented as a plurality of tiles and the tiled images are used for further processing), and then perform brain imitation visual recognition on the divided tile-unit images to determine to reduce a DNN inference range required for 3D rendering (removing useless DNN INFs is functionally equivalent to reducing a DNN inference range; Figure 2.7.2 shows the VPC comprises a TTH (tile-wise TF Handler); page 50 col. 1 paragraph 5: Firstly, the VPC performs 3 visual perception stages of BuFF to determine the workload of the HNEs after removing useless DNN INFs; page 50 col. 1 paragraph6: Figure 2.7.3 shows the details of the TF Unit (TFU) and SA Unit (SAU) in the VPC. The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH). The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA). The TTH selects only one RGB-D pixel closest to the camera origin using a depth comparator during intra-tile handling);
a hybrid neural engine (HNE) (this element is interpreted as hardware under 35 USC 112(f); page 50 col. 1 paragraph 4: The proposed MetaVRain has 3 key hardware blocks: 2) 1D–2D Hybrid Neural Engines (HNE) using Dynamic Neural Network Allocation (DNNA) for fast and efficient DNN processing) including a plurality of neural engines (NEs) (Figure 2.7.2 shows a plurality of HNEs) having different operational efficiencies according to input sparsity and configured to accelerate DNN inference included in the DNN inference range determined to be reduced by dividing and allocating tasks each having reduced operation quantity for the DNN inference to the plurality of NEs (dynamically allocating tasks to 1D and 2D NEs is functionally analogous to dividing and allocating tasks; page 50 col. 1 paragraph 5: The remaining DNN INFs are accelerated by HNEs and their tasks are dynamically allocated to the two heterogeneous engines, 1D and 2D NEs, by the CS-DNNA Core. The 1D NE consists of 8 PE units and each PE unit receives a single nonzero (NZ) input activation (IA) to compute 32 outputs simultaneously. It maximizes zero skipping but sacrifices data reuse, resulting in significant performance degradation if the sparsity ratio is <50%. On the other hand, the 2D NE maximizes data reuse to be more favorable for dense inputs. It receives 32 IAs from I/O memory (IOMEM) or the PEU regardless of sparsity and generates 32 outputs using adder trees. All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to IOMEM after post-processing, such as ReLU; page 50 col. 2 paragraph 1: The Task Controller manages the I/O data for the HNE with out-of-order execution (OoOE); page 50 col. 2 paragraph 2: The HNE utilizes DNNA to divide DNN channels into two groups according to the sparsity ratio. The 1D NE receives channels where sparsity is higher than a threshold, and the remaining channels are allocated to the 2D NE. The CS-ONNA Core predicts the sparsity of the next layer with CS, which searches the 4 outermost empty pixels of every tile in zigzag order from center to the periphery of 4 sub-tiles. The selected 4 pixels are used to perform 8-sample INFs by the HNE and generate the CS-Bmap indicating NZ outputs. They are aggregated by ‘OR’ operations and generate a CS bit-mask for every input channel. The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask. The efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone); and
a dynamic neural network allocation (DNNA) core (this element is interpreted as hardware under 35 USC 112(f); Figure 2.7.2 shows CS-DNNA core; page 50 col. 1 paragraph 4: The proposed MetaVRain has 3 key hardware blocks: 2) 1D–2D Hybrid Neural Engines (HNE) using Dynamic Neural Network Allocation (DNNA) for fast and efficient DNN processing) configured to generate selection information for allocating each of the tasks to one of the plurality of NEs based on a sparsity ratio of each of the tile-unit images (Figure 2.7.4 shows Tile-wise polarized sparsity of an image; page 50 col. 2 paragraph 2: The HNE utilizes DNNA to divide DNN channels into two groups according to the sparsity ratio. The 1D NE receives channels where sparsity is higher than a threshold, and the remaining channels are allocated to the 2D NE. The CS-ONNA Core predicts the sparsity of the next layer with CS, which searches the 4 outermost empty pixels of every tile in zigzag order from center to the periphery of 4 sub-tiles. The selected 4 pixels are used to perform 8-sample INFs by the HNE and generate the CS-Bmap indicating NZ outputs. They are aggregated by ‘OR’ operations and generate a CS bit-mask for every input channel. The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask. The efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone).
Regarding claim 11, Han teaches an AI-based high-speed and low-power 3D rendering method using a 3D rendering accelerator (page 50 col. 2 paragraph 5: MetaVRain is fabricated in 28nm CMOS technology, integrating 5K FP8-FP16 configurable MACs with 2MB of SRAM. Its power and throughput can be varied dynamically by selecting operating modes with SW programming, and there are two typical operating modes: power-efficient and high-speed. In power-efficient mode, the chip consumes 133mW while maintaining >30fps. In high-speed mode, it can achieve a maximum of 118fps with 899mW power consumption. In summary, we present an efficient NeRF architecture, BuFF, and we design and fabricate MetaVRain with the proposed BuFF architecture for low-power and real-time 3D rendering) based on a DNN trained using a weight of the DNN using a plurality of 2D photos obtained by imaging the same object from several directions and then configured to perform 3D rendering using the same (Figure 2.7.1 “Overview of NeRF w/3 Main Processes” shows the process using a plurality of images of an object as seen from different VR head positions (different viewing directions) and shows that the volumetric rendering is performed using weighted-sum of the DNN INF; page 50 col. 1 paragraph 1: A neural radiance field (NeRF) [1] uses a deep neural network (DNN) to create 3D models by training the DNN to memorize 3D scene geometry from a few photos), the 3D rendering method comprising:
creating an image plane for an object that is a 3D rendering target from a position and a direction of an observer (Figure 2.7.1 shows an image plane of the viewed object is formed by the position and direction of the VR user), dividing the image plane into a plurality of tile and units (it is inherent that the image as displayed in an image plane is divided into a plurality of tile units for Tile-wise 3D Transform and Tile-wise polarized sparsity; Figure 2.7.3 shows Tile-wise 3D Transform of an image represented by a plurality of tiles; Figure 2.7.4 shows Tile-wise polarized sparsity where an image is represented as a plurality of tiles and the tiled images are used for further processing), then performing brain imitation visual recognition on the divided tile-unit images to determine to reduce a DNN inference range required for 3D rendering (removing useless DNN INFs is functionally equivalent to reducing a DNN inference range; Figure 2.7.2 shows the VPC comprises a TTH (tile-wise TF Handler); page 50 col. 1 paragraph 5: Firstly, the VPC performs 3 visual perception stages of BuFF to determine the workload of the HNEs after removing useless DNN INFs; page 50 col. 1 paragraph6: Figure 2.7.3 shows the details of the TF Unit (TFU) and SA Unit (SAU) in the VPC. The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH). The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA). The TTH selects only one RGB-D pixel closest to the camera origin using a depth comparator during intra-tile handling);
determining a DNN reference method of each of the tile-unit images based on a sparsity ratio of each of the tile-unit images by dividing and allocating operation tasks of DNN inference included in the DNN inference range to a plurality of NEs (Figure 2.7.2 shows a plurality of HNEs) having different operational efficiencies according to input sparsity (dynamically allocating tasks to 1D and 2D NEs is functionally analogous to dividing and allocating tasks; Figure 2.7.4 shows Tile-wise polarized sparsity of an image; page 50 col. 1 paragraph 5: The remaining DNN INFs are accelerated by HNEs and their tasks are dynamically allocated to the two heterogeneous engines, 1D and 2D NEs, by the CS-DNNA Core. The 1D NE consists of 8 PE units and each PE unit receives a single nonzero (NZ) input activation (IA) to compute 32 outputs simultaneously. It maximizes zero skipping but sacrifices data reuse, resulting in significant performance degradation if the sparsity ratio is <50%. On the other hand, the 2D NE maximizes data reuse to be more favorable for dense inputs. It receives 32 IAs from I/O memory (IOMEM) or the PEU regardless of sparsity and generates 32 outputs using adder trees. All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to IOMEM after post-processing, such as ReLU; page 50 col. 2 paragraph 1: The Task Controller manages the I/O data for the HNE with out-of-order execution (OoOE); page 50 col. 2 paragraph 2: The HNE utilizes DNNA to divide DNN channels into two groups according to the sparsity ratio. The 1D NE receives channels where sparsity is higher than a threshold, and the remaining channels are allocated to the 2D NE. The CS-ONNA Core predicts the sparsity of the next layer with CS, which searches the 4 outermost empty pixels of every tile in zigzag order from center to the periphery of 4 sub-tiles. The selected 4 pixels are used to perform 8-sample INFs by the HNE and generate the CS-Bmap indicating NZ outputs. They are aggregated by ‘OR’ operations and generate a CS bit-mask for every input channel. The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask. The efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone); and
accelerating the DNN inference using the DNN reference method determined in the determining a DNN reference method (page 50 col. 1 paragraph 5: The remaining DNN INFs are accelerated by HNEs and their tasks are dynamically allocated to the two heterogeneous engines, 1D and 2D NEs, by the CS-DNNA Core. The 1D NE consists of 8 PE units and each PE unit receives a single nonzero (NZ) input activation (IA) to compute 32 outputs simultaneously. It maximizes zero skipping but sacrifices data reuse, resulting in significant performance degradation if the sparsity ratio is <50%. On the other hand, the 2D NE maximizes data reuse to be more favorable for dense inputs. It receives 32 IAs from I/O memory (IOMEM) or the PEU regardless of sparsity and generates 32 outputs using adder trees. All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to IOMEM after post-processing, such as ReLU; page 50 col. 2 paragraph 2: The HNE utilizes DNNA to divide DNN channels into two groups according to the sparsity ratio. The 1D NE receives channels where sparsity is higher than a threshold, and the remaining channels are allocated to the 2D NE. The CS-ONNA Core predicts the sparsity of the next layer with CS, which searches the 4 outermost empty pixels of every tile in zigzag order from center to the periphery of 4 sub-tiles. The selected 4 pixels are used to perform 8-sample INFs by the HNE and generate the CS-Bmap indicating NZ outputs. They are aggregated by ‘OR’ operations and generate a CS bit-mask for every input channel. The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask. The efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2-10 and 12-20 is/are rejected under 35 U.S.C. 103 as being obvious over Han, and further in view of Han et al. (“MetaVRain: A Mobile Neural 3-D Rendering Processor With Bundle-Frame-Familiarity-Based NeRF Acceleration and Hybrid DNN Computing”, hereinafter Han2).
The applied reference of Han2 has common inventors and/or a common applicant/assignee with the instant application. Based on the publication date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(1). However, the instant application claims a benefit of foreign priority under 35 U.S.C. 119(a)-(d) that antedates the publication date of Han2, but no English language translation of a non-English language foreign application is provided in accordance with 37 CFR 1.55. The filing date of the priority document is not perfected unless applicant has filed a certified priority document in the application (and an English language translation, if the document is not in English) (see 37 CFR 1.55(g)). See MPEP 2152.06(C).
Applicant cannot rely upon the certified copy of the foreign priority application to overcome this rejection because a translation of said application has not been made of record in accordance with 37 CFR 1.55. When an English language translation of a non-English language foreign application is required, the translation must be that of the certified copy (of the foreign application as filed) submitted together with a statement that the translation of the certified copy is accurate. See MPEP §§ 215 and 216.
Regarding claim 2, Han teaches the 3D rendering accelerator according to claim 1, wherein the VPC comprises:
a temporal familiarity unit (TFU) (this element is interpreted as hardware under 35 USC 112(f); TF Unit (TFU), Figure 2.7.3; page 50 col. 1 paragraph 5: Figure 2.7.3 shows the details of the TF Unit (TFU) and SA Unit (SAU) in the VPC) configured to determine a reused pixel value (evaluate pixel data can be reused), which is a pixel value of a previous frame allowed to be reused, by evaluating familiarity of the previous frame and a current frame (Figure 2.7.1 shows a plurality of positions of the observer viewing the object; page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration; page 50 col. 1 paragraph 6: The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused), determine DNN inference target pixels that require DNN inference among pixels of the current frame in consideration of the reused pixel value, and then generate a binary map (page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration; page 50 col. 1 paragraph 6 – page 50 col. 2 paragraph 1: The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH). The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA). The TTH selects only one RGB-D pixel closest to the camera origin using a depth comparator during intra-tile handling. In the inter-tile handling, it skips the depth comparison but, by a bitwise ‘AND’ operation, determines overlapping pixels that need a new INF. The TFU reduces EMA by 99.5% and enables the VPC to reuse RGB-D pixels of the previous frame, resulting in 48.6× speed-up, on average … the binary map (Bmap) received from the TFU); and
a spatial attention unit (SAU) (this element is interpreted as hardware under 35 USC 112(f); TF Unit (TFU), Figure 2.7.3; page 50 col. 1 paragraph 5: Figure 2.7.3 shows the details of the TF Unit (TFU) and SA Unit (SAU) in the VPC)) configured to express the object using low-resolution voxels generated (page 50 col. 1 paragraph 3: The SA stage utilizes an attention map with low-resolution voxels to collect only meaningful samples inside the attention map), (page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input. The Voxel Cache uses the coordinates to fetch new voxels only in the case of cache misses. The voxel is used to evaluate whether the sample is inside the attention map or not, and then the tile is transferred to the either the dense or sparse buffer of the Task Controller, depending on the number of valid samples per tile).
Han does not explicitly teach the previous frame and a current frame corresponds to the observer changing positions; a binary map representing the DNN inference target pixels; a spatial attention unit (SAU) configured to express the object using low-resolution voxels generated in advance; then exclusively generate coordinates located inside the voxels in a gaze direction of the observer as sample coordinates subjected to DNN inference.
Han2 teaches the previous frame and a current frame corresponds to the observer changing positions (fig. 7 shows moving the camera viewpoints changes the position of the observer and generates a first frame from a first position and a next frame from a second position); a binary map representing the DNN inference target pixels (pixels that are reusable are functionally analogous to target pixels; page 69 col. 1 paragraph 2: As shown in Fig. 8, the TFU transfers the transformation results to the HNEs and it calculates the familiarity using the partial INF. To minimize the overhead of the partial INF, it loads intermediate feature maps of the previous pixel and updates the color value using the updated view direction of the current frame. Since it reuses the previous INF results by loading the feature map, the partial INF shows only 0.72% computation amount compared with full INF. When the HNE completes the partial INF, it makes a binary map (Bmap) to indicate which pixels are reusable); a spatial attention unit (SAU) configured to express the object using low-resolution voxels generated in advance (page 67 col. 1 last paragraph – page 67 col. 2 first paragraph: the proposed SA stage successfully reduces the number of samples while maintaining the rendering quality. Before it starts the rendering, it predefined low-resolution voxel to represent a 3-D model. This voxel can be obtained during the pretraining phase by indicating the sub-spaces which contain the high-density samples. After voxel acquisition, it follows the sampling method based on Fig. 4(b) but skips sampling outside the attention map. In other words, it collects only the meaningful samples inside the low-resolution voxel); then exclusively generate coordinates located inside the voxels in a gaze direction of the observer as sample coordinates subjected to DNN inference (shooting a ray and determining the samples lying on the same ray is functionally analogous to samples located inside the voxels in a gaze direction of the observer; page 70 col. 1 paragraph 2: The SAU consists of a sample coordinate generator (SCG), a 4-kB voxel cache, and a task controller. Since the TFU selects only a portion of pixels that need new RGB-D values, the role of SAU is to generate sample coordinates of the corresponding pixels. During the sample coordinate generation, the SAU shoots a ray based on the accumulated Bmap received from the TFU. It finds out the samples lying on the same ray while judging whether the samples are placed within the attention map; page 70 col. 1 paragraph 2: The SAU generates coordinates of the 8 samples per pixel by evaluating whether the sample is inside the attention map or not. Then, a tile is classified as dense or sparse tile depending on the number of valid samples per tile). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Han2’s knowledge of using a VPC with a temporal familiarity unit and a spatial attention unit as taught and modify the system of Han because such a system provides a low-power neural 3D rendering processor for metaverse realization on mobile devices (abstract) that resolves the high computational cost problem and avoids performance degradation (page 72 col. 2 paragraph VI. Modulo-base PEU).
Regarding claim 3, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 2, wherein the TFU comprises:
a 3D transformer (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 1 last paragraph: Figure 2.7.3 shows the details of the TF Unit (TFU) and SA Unit (SAU) in the VPC. The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH)) configured to perform 3D transformation on the tile-unit images (Han – page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation; Han – page 50 col. 1 last paragraph: The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA)), then rotate and parallel-transfer RGB values for each pixel of the previous frame based on change values of the position and the direction of the observer, and project the RGB values onto corresponding pixels of the current frame (rotate and parallel-transfer of RGB values for each pixel of the previous frame is achieved by using the relative addressing-based buffering; Han – page 50 col. 1 last paragraph: The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA); Han2 – page 69: paragraph 1: it projects the previous RGB-D pixels into the current frame via 3-D transformation. As shown in Fig. 7, since the pixel position after transformation cannot be predicted in advance, the 3-D transformation usually requires a large output buffer to cover all destinations of the pixels; Han2 – page 69 paragraph 2: Through the 3-D transformation, the MetaVRain can find candidates of the reusable pixels, but as mentioned in Section II, the TFU does not directly reuse the RGB-D values of the previous frame. It performs TF evaluation after 3-D transformation to measure the familiarity of the color. As shown in Fig. 8, the TFU transfers the transformation results to the HNEs and it calculates the familiarity using the partial INF); and
a reuse processor (TFU) configured to predict a color change value in units of pixels of the previous frame and the corresponding current frame, and determine to exclusively reuse an RGB value of a pixel in which the predicted color change value is less than a preset allowable change value (comparing color differences before and after transformation and determining if the color difference is small inherently involves comparing the color difference to a threshold or predefined value; Han - page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration; Han2 – page 67 col. 2 last paragraph – page 68 col. 1 first paragraph: The TF evaluation is a step to predict the color differences before and after transformation. It estimates color changes by comparing DNN INF results. Although the density is constant, the color of a sample is varied according to the camera coordinates. Since the intermediate feature map that appeared during the density calculation is not changed even with the varying camera positions, our TF evaluation estimates the color changes by analyzing only the last few layers which are affected by the camera view direction. In the previous frame calculation, it remembers the input feature map of the camera-pose-dependent layers and performs “partial INF” only for the remaining layers. If the TF evaluation finds out that a pixel of the previous frame has less color change, then, the corresponding RGB-D pixel can be reused in the current frame rendering due to its temporally high familiarity. On the contrary, if familiarity is low, it calculates a new RGB-D value through a new INF. Although the naïve RGB-D reuse without TF evaluation shows 9.7 dB PSNR loss, the proposed TF can successfully skip more than 95% of DNN INFs with only less than 1 dB PSNR loss; page 69 col. 1 paragraph 2: the TFU does not directly reuse the RGB-D values of the previous frame. It performs TF evaluation after 3-D transformation to measure the familiarity of the color. As shown in Fig. 8, the TFU transfers the transformation results to the HNEs and it calculates the familiarity using the partial INF. To minimize the overhead of the partial INF, it loads intermediate feature maps of the previous pixel and updates the color value using the updated view direction of the current frame).
Regarding claim 4, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 3, wherein the 3D transformer rotates and parallel-transfers the RGB values for each pixel of the previous frame by utilizing a relative addressing-based buffering technique (Han – page 50 col. 1 last paragraph: The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA); Han2 – page 69: paragraph 1: it projects the previous RGB-D pixels into the current frame via 3-D transformation. As shown in Fig. 7, since the pixel position after transformation cannot be predicted in advance, the 3-D transformation usually requires a large output buffer to cover all destinations of the pixels; Han2 – page 69 paragraph 2: Through the 3-D transformation, the MetaVRain can find candidates of the reusable pixels, but as mentioned in Section II, the TFU does not directly reuse the RGB-D values of the previous frame. It performs TF evaluation after 3-D transformation to measure the familiarity of the color. As shown in Fig. 8, the TFU transfers the transformation results to the HNEs and it calculates the familiarity using the partial INF).
Regarding claim 5, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 3, wherein, when an overlap phenomenon occurs in which a plurality of previous frame pixels matches one current frame pixel (Han – page 50 col. 1 last paragraph: In the inter-tile handling, it skips the depth comparison but, by a bitwise ‘AND’ operation, determines overlapping pixels that need a new INF; Han2 – page 69 col. 1 paragraph 3” After 3-D transformation, multiple RGB-D pixels can be located on the same ray. This overlapping problem occurs within a tile (Intra-tile) or among the tiles (Inter-tile). Although the intra-tile overlapping can be discovered in a single calculation phase, the inter-tile overlapping can be found after a few calculation phases), the reuse processor solves intra-tile overlap by selecting an RGB value of a pixel located at a frontmost side among the previous frame pixels where overlap occurs and solves inter-tile overlap by DNN inference (Han2 – page 69 col. 2 last paragraph: Therefore, the built-in TTH deals with this problem by adopting two different ways. During the intra-tile handling, it selects only one RGB-D pixel closest to the camera origin using a depth comparator. On the contrary, the TTH skips depth comparison in the inter-tile handling. Instead, it finds out the overlapping pixels using the bitwise “AND” operation. The Bmap generated by the “AND” operation responds to the transformation results of adjacent tiles and the final accumulated Bmap indicates whether it needs an INF to get a new RGB-D value).
Regarding claim 6, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 2, wherein the SAU comprises:
a voxel cache (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller) configured to store information about the voxels collected from an external memory (Han2 – page 70 col. 1 paragraph 2: The loading of the attention map is done by the built-in voxel cache to minimize EMA; it should be noted that EMA stands for external memory access, Han2 – page 69 col. 1 paragraph 1);
a sample coordinate generator (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller) configured to exclusively generate coordinates inside the voxels corresponding to the DNN inference target pixels among coordinates on a ray as sample coordinates by emitting the ray based on the binary map (Han – page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input; Han2 – page 70 col. 1 paragraph 2: the role of SAU is to generate sample coordinates of the corresponding pixels. During the sample coordinate generation, the SAU shoots a ray based on the accumulated Bmap received from the TFU. It finds out the samples lying on the same ray while judging whether the samples are placed within the attention map. It can stop the calculation of the sample coordinates by receiving the accumulated density from TDA logic. Thanks to the help of TDA logic, the SAU can avoid the sample-coordinate generation of useless samples. The loading of the attention map is done by the built-in voxel cache to minimize EMA. It fetches only the new voxels using the cache misses, which minimizes the repeated voxel loading); and
a task controller (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller) configured to determine a number of effective samples (number of valid samples per tile) for each of the DNN inference target pixels based on a result of sample coordinate generation, use a result thereof to classify the tile-unit images into a dense tile or a sparse tile, and then control input/output (input controller and output controller) to/from the HNE according to a type thereof (Han – page 50 col. 2 first paragraph: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input. The Voxel Cache uses the coordinates to fetch new voxels only in the case of cache misses. The voxel is used to evaluate whether the sample is inside the attention map or not, and then the tile is transferred to the either the dense or sparse buffer of the Task Controller, depending on the number of valid samples per tile. The Task Controller manages the I/O data for the HNE with out-of-order execution (OoOE). The input controller transfers dense tiles directly to the HNE, but in the case of sparse tiles, it waits until input coordinates are fully accumulated in a buffer. The output controller decodes the HNE output with either the Masking Unit or Reorder Buffer (ROB) to generate final RGB-D results.; Han2 – page 70 col. 1 last paragraph: Since the efficiency of the HNE can be further improved when enough samples exist in a tile, the task controller manages the I/O data of the HNE with out-of-order execution (OoOE) as described in Fig. 9. The input controller transfers dense tiles directly to the HNE, but in the case of sparse tiles, it waits until input coordinates are accumulated enough in a buffer. The output controller decodes HNE output using either the masking unit or the reorder buffer (ROB) to generate final RGB-D results).
Regarding claim 7, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 6, wherein the SAU further comprises top-down attention (TDA) logic (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 1 paragraph 3: It consists of 3 visual perception stages: 1) Spatial Attention (SA), 2) Temporal Familiarity (TF), and 3) Top-Down Attention (TDA)) configured to determine whether to continue inference based on a DNN inference result and control a sample coordinate generation operation of the sample coordinate generator (Han – page 50 col. 1 paragraph 3: The TDA stage skips the last few layers of the DNN when the density obtained from the prior layers is low (short-term feedback). It also skips computing the remaining samples along a ray when the cumulative density of prior samples is high enough (long-term feedback); Han – page 50 col. 2 first paragraph: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input; Han2 – page 68 col. 1 paragraph 2: The last stage, TDA, receives intermediate or final DNN INF results as feedback to determine whether it continues or skips the remaining operations. In the short-term feedback, it calculates density using the prior DNN layers. And then if the obtained density is low enough, it skips the calculation of the last few layers because it means that the sample is transparent. Although short-term feedback utilizes the intermediate density value, long-term feedback utilizes accumulated density as feedback. If the cumulative density is high enough, the results of the remaining samples do not affect the final result. The long-term feedback helps the SA stage and skips the computing of the remaining samples along a ray when the cumulated density of prior samples is high enough),
wherein the TDA logic is configured to:
receive an intermediate operation result for any first layer from the HNE, and determine to skip remaining DNN inference steps of the first layer when an intermediate density value included in the intermediate operation result is less than a preset downward threshold (Han – page 50 col. 1 paragraph 3: The TDA stage skips the last few layers of the DNN when the density obtained from the prior layers is low (short-term feedback); Han2 – page 68 col. 1 paragraph 2: The last stage, TDA, receives intermediate or final DNN INF results as feedback to determine whether it continues or skips the remaining operations. In the short-term feedback, it calculates density using the prior DNN layers. And then if the obtained density is low enough, it skips the calculation of the last few layers because it means that the sample is transparent), and
receive a cumulative density value generated throughout a 3D rendering process from the HNE, and determine to suspend the sample coordinate generation operation of the sample coordinate generator when the cumulative density value exceeds a preset upward threshold (Han – page 50 col. 2 first paragraph: It also skips computing the remaining samples along a ray when the cumulative density of prior samples is high enough (long-term feedback); Han2 – page 68 col. 1 paragraph 2: The last stage, TDA, receives intermediate or final DNN INF results as feedback to determine whether it continues or skips the remaining operations … Although short-term feedback utilizes the intermediate density value, long-term feedback utilizes accumulated density as feedback. If the cumulative density is high enough, the results of the remaining samples do not affect the final result. The long-term feedback helps the SA stage and skips the computing of the remaining samples along a ray when the cumulated density of prior samples is high enough).
Regarding claim 8, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 2, wherein the HNE comprises:
a 1D NE (this element is interpreted as hardware under 35 USC 112(f); Han – page 5 col. 1 paragraph 4: 1D-2D Hybrid Neural Engines (HNE)) configured to skip a corresponding operation when there is 0 in input data (Han – page 50 col. 1 paragraph 5: The 1D NE consists of 8 PE units and each PE unit receives a single nonzero (NZ) input activation (IA) to compute 32 outputs simultaneously. It maximizes zero skipping but sacrifices data reuse, resulting in significant performance degradation if the sparsity ratio is <50%; Han – page 50 col. 2 paragraph 2: The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask; Han2 – page 68 col. 1 last paragraph: The 1-D NE maximizes zero skipping ability but sacrifices data-reusing);
a 2D NE (this element is interpreted as hardware under 35 USC 112(f); Han – page 5 col. 1 paragraph 4: 1D-2D Hybrid Neural Engines (HNE)) configured to reuse data regardless of whether 0 is present in input data (Han – page 50 col. 1 paragraph 5: the 2D NE maximizes data reuse to be more favorable for dense inputs. It receives 32 IAs from I/O memory (IOMEM) or the PEU regardless of sparsity and generates 32 outputs using adder trees. All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to IOMEM after post-processing, such as ReLU; Han – page 50 col. 2 paragraph 2: The CS-ONNA Core prefetches weights from global memory into either the 1D or 2D NE according to the number of zeros in the bit-mask; Han2 – page 68 col. 1 last paragraph: the 2-D NE adopts an architecture that maximizes data reuse, which is more favorable for dense inputs);
an input/output memory IOMEM (this element is interpreted as hardware under 35 USC 112(f); Han – page 50 col. 1 paragraph 5: I/O memory (IOMEM)) configured to store input/output values of the 1D NE and the 2D NE (Han – page 50 col. 1 paragraph 5: It receives 32 IAs from I/O memory (IOMEM) … All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to IOMEM after post-processing, such as ReLU; Han2 – page 68 col. 2 paragraph 1: The 2-D NE receives 32 IAs from I/O memory (IOMEM) regardless of sparsity … All intermediate partial-sums are stored in partial-sum memory (PSMEM) and transferred to the IOMEM after post-processing such as ReLU … Both 1-D and 2-D NE can receive IA from not only IOMEM but also a positional encoding unit (PEU) to calculate the layer which needs positional encoding results); and
a modulo-based positional encoding unit (Mod-PEU) (this element is interpreted as hardware under 35 USC 112(f); Han – page 5 col. 1 paragraph 4: Modulo-based Positional Encoding Unit (Mod-PEU) to minimize the HW cost of the sinusoidal function necessary for NeRF operation) configured to generate a sine wave using an approximation formula including a quadratic function and a modulo function and perform positional encoding using the sine wave (Han – page 5 col. 1 paragraph 4: Modulo-based Positional Encoding Unit (Mod-PEU) to minimize the HW cost of the sinusoidal function necessary for NeRF operation; Han – page 50 col. 2 paragraph 3: Figure 2.7.5 shows detailed circuits for the Mod-PEU adopting periodic polynomial approximation of a sinusoidal function. It consists of a Modulo Circuit, Sign Calculation Circuit (SCC), and a Magnitude Calculation Circuit (MCC); Han2 – page 72 col. 2 paragraph 3: As shown in Fig. 17, the MetaVRain approximates sinusoidal functions with the simple periodic polynomials during the positional encoding. The polynomial consists of quadratic functions and modulo functions, and due to the modulo calculation, it has repeated outputs with period 4. This approximation can be used directly without fine-tuning but with fine-tuning, we can get exactly the same 3-D rendering quality. The reason why it maintains the same rendering quality after fine-tuning is that this approximation method is still differentiable and the direction of its gradient is almost the same as the sinusoidal function; Han2 – page 72 – col. 2 last paragraph - page 73 col. 1 first paragraph: Figs. 18 and 19 show the detailed circuits of Mod-PEU after adopting periodic polynomial approximation. It consists of a modulo circuit, a sign calculation circuit (SCC), and a magnitude calculation circuit (MCC). The modulo circuit makes two modulo residues: mod(x,2) and mod(2−x,2) . It receives the coordinates of the sample with the IEEE 754 half-precision format and obtains mod(x,2) with the simple arithmetic bit-shifter. After that, mod(2−x,2) can be simply calculated with the 2’s complement unit because it has the same value as the mod(−x,2)).
Regarding claim 9, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 8, wherein the Mod-PEU calculates a remainder of dividing sample coordinates in half-precision format by 2, takes 2's complement thereof to generate first and second parameters, and then applies the first and second parameters to an equation below to generate a plurality of sine waves and cosine waves required for positional (Han2 – page 72 col. 2 paragraph 3: As shown in Fig. 17, the MetaVRain approximates sinusoidal functions with the simple periodic polynomials during the positional encoding. The polynomial consists of quadratic functions and modulo functions, and due to the modulo calculation, it has repeated outputs with period 4. This approximation can be used directly without fine-tuning but with fine-tuning, we can get exactly the same 3-D rendering quality. The reason why it maintains the same rendering quality after fine-tuning is that this approximation method is still differentiable and the direction of its gradient is almost the same as the sinusoidal function; Han2 – page 72 col. 2 last paragraph - page 73 col. 1 first paragraph: Figs. 18 and 19 show the detailed circuits of Mod-PEU after adopting periodic polynomial approximation. It consists of a modulo circuit, a sign calculation circuit (SCC), and a magnitude calculation circuit (MCC). The modulo circuit makes two modulo residues: mod(x,2) and mod(2−x,2) . It receives the coordinates of the sample with the IEEE 754 half-precision format and obtains mod(x,2) with the simple arithmetic bit-shifter. After that, mod(2−x,2) can be simply calculated with the 2’s complement unit because it has the same value as the mod(−x,2); Han2 – page 73 col. 1 paragraph 2: The positional encoding requires multiple encoding results with the “power of 2” sequence. Both the proposed SCC and MCC make 30 positional encoding values in parallel by considering this characteristic. As shown in Fig. 18, the sign of the sine functions can be determined using the XOR operation between the MSB bits of the arithmetic bit-shifting results and a sign bit of the original data. On the other hand, it simply adds 2’b01 to MSB bits to generate the sign of the cosine functions; Han2 – page 73 col. 1 paragraph 3: The remaining exponent and mantissa of the sinusoidal function are calculated by the MCC. It multiplies mod(x,2) and mod(2−x,2) to make the magnitudes of the sine functions. On the contrary, cosine functions require multiplication results of the mod(x+1,2) and mod(1−x,2) . During the magnitude calculation through the MCC, we found that both sine and cosine functions require the same LSB multiplication results. Moreover, the LSB multiplication can be reused to make multiple positional encoding results when considering the power of 2 multiplication. For this reason, the MCC shares the LSB-part multiplier during the generation of multiple encoding results and further reduces 38.2% power consumption) encoding:
(Equation)
sin(2-1πx) ≈ (-1)[x/2] .mod(x,2).mod(2-x,2) (Han – Figure 2.7.5 shows this equation; Han2 – fig. 17)
cos(2-1πx) ≈ (-1)[(x+1)/2] . mod(x+1,2).mod(1-x,2) (Han – Figure 2.7.5 shows this equation; Han2 – fig. 17).
Regarding claim 10, the combination of Han and Han2 teaches the 3D rendering accelerator according to claim 2, wherein the DNNA core comprises:
an output zero pattern extractor (this element is interpreted as hardware under 35 USC 112(f); CS-DNNA Core, Figure 2.7.4 - Han) configured to determine some of the DNN inference target pixels as sparsity reference pixels for each of the tile-unit images using a centrifugal sampling (CS) method, and extract an output zero pattern of each of the sparsity reference pixels (Han2 – page 71 col. 1 paragraph 1: The CS-DNNA core collects the sparsity of a small portion of samples in a tile and from this, it predicts the sparsity patterns of the other remaining samples. Since a tile consists of 64 pixels and the CS collects sparsity patterns of the 4 pixels per tile, the sparsity prediction results can be reused for maximally 60 pixels. Instead, the CS-DNNA core considers the worst case of sparsity received from four representative pixels;);
a bitmask generator (this element is interpreted as hardware under 35 USC 112(f); CS-DNNA Core comprising Bit-mask, Figure 2.7.4 - Han) configured to collect an output zero pattern for each of the sparsity reference pixels using a bitwise OR operation to generate a bitmask representing the sparsity ratio (Han2 – page 71 col. 1 paragraph 2: In each sub-tile, the CS searches an empty pixel in zigzag order from the center to the periphery. This zigzag-searching pattern shows a 90° rotated form compared with the adjacent sub-tiles. After 4-pixels-selection, the HNE accelerates INFs of 8 samples for each pixel and generates the CS-Bmap which indicates NZ outputs. The CS-DNNA core judges the worst case of zero patterns by aggregating the CS-Bmaps using the “OR” operation. This aggregation result is called a “CS bit mask” and it is generated for every input channel);
a selection information generator (this element is interpreted as hardware under 35 USC 112(f); CS-DNNA Core, Figure 2.7.4 - Han) configured to generate selection information for allocating each of the tasks to one of the plurality of NEs according to a number of 0s in the bitmask (Han2 - page 71 col. 1 last paragraph – page 71 col. 2 first paragraph: The HNE utilizes the DNNA to divide DNN channels into two groups according to the sparsity ratio. In this process, the CS-DNNA core prefetches weights from global memory into either 1-D or 2-D NE according to the number of zeros in the CS bit mask. The 1-D NE receives channels of which sparsity is higher than 62.5% sparsity, and the remaining channels are allocated to the 2-D NE; Han2 – page 72 col. 1 paragraph 1: The efficiency of the HNEs can be further improved using the prediction results of the CS-DNNA core. After generating the CS bit mask, the HNE can figure out operations that do not need to be computed because a portion of outputs has already been determined as zero value. Thus, the HNE is designed with a clock-gating (CG) circuit which consists of bit-rotators. As shown in Fig. 15, the CG circuits are placed in both 1-D and 2-D NEs, and they receive the CS bit mask as an input signal. The bit-rotators make CG control signals to manage the power consumption of the pipeline structure in the main PE arrays. Fig. 16 explains details about the CG circuit operation. After loading a bit-mask, the bit-mask should be rotated to consider the next sample);
an offloading controller (this element is interpreted as hardware under 35 USC 112(f); CS-DNNA Core, Figure 2.7.4 - Han) configured to monitor task allocation information of each of the plurality of NEs according to the selection information, perform task offloading according to an operation execution time occupancy characteristic of each of the plurality of NEs, and perform offloading to improve utilization of one NE having constant operation execution time occupancy among the plurality of Nes (Han – page 50 col. 2 paragraph 2: he efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone; Han2 – page 71 col. 2 last paragraph – page 72 col. 1 first paragraph - the CS-DNNA core detects underutilization of the 2-D NE in advance, then, it performs either 1-D-to-2-D or 2-D-to-1-D task offloading to maintain 2-D NE with full core utilization status. In Fig. 14, we examined the CS-based task-offloading in both SND and the forward-facing dataset (FFD [2])); and
a clock controller (this element is interpreted as hardware under 35 USC 112(f); CS-base CG (clk-gating), Figure 2.7.4 - Han) configured to control a clock of each of the plurality of NEs based on the bitmask (Han – page 50 col. 2 paragraph 2: he efficiency of the HNE can be further improved by both task-offloading and clock-gating (CG). Since the CS-ONNA Core can detect underutilization of the 2D NE in advance, it realizes both 10-to-20 and 20-to-1 Dtask-offloading to improve throughput by 14.5%. Furthermore, bit-rotator-based CG circuits in the 1D and 2D NEs use the CS bit-mask to generate CG signals and reduce the dynamic power of PE operation by up to 24.6%. The HNE achieves 3.7× higher throughput than acceleration without exploiting sparsity and at least 2.4× higher energy efficiency compared with using solely a 1D or 2D NE alone; Han2 – page 72 col. 1 paragraph 1: the HNE is designed with a clock-gating (CG) circuit which consists of bit-rotators. As shown in Fig. 15, the CG circuits are placed in both 1-D and 2-D NEs, and they receive the CS bit mask as an input signal. The bit-rotators make CG control signals to manage the power consumption of the pipeline structure in the main PE arrays. Fig. 16 explains details about the CG circuit operation. After loading a bit-mask, the bit-mask should be rotated to consider the next sample. After it completes the last sample, it starts the new 8-sample INFs of the nearby pixel. The CS-based CG finally reduces the dynamic power consumption of the HNEs by up to 24.6%).
Regarding claim 12, Han teaches the 3D rendering method according to claim 11, wherein the creating comprises:
expressing the object using low-resolution voxels generated (page 50 col. 1 paragraph 3: The SA stage utilizes an attention map with low-resolution voxels to collect only meaningful samples inside the attention map), and then exclusively generating coordinates located inside the voxels (page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input. The Voxel Cache uses the coordinates to fetch new voxels only in the case of cache misses. The voxel is used to evaluate whether the sample is inside the attention map or not, and then the tile is transferred to the either the dense or sparse buffer of the Task Controller, depending on the number of valid samples per tile);
determining a reused pixel value (evaluate pixel data can be reused), which is a pixel value of a previous frame allowed to be reused, by evaluating familiarity of the previous frame and a current frame (Figure 2.7.1 shows a plurality of positions of the observer viewing the object; page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration; page 50 col. 1 paragraph 6: The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused), and determining DNN inference target pixels that require DNN inference among pixels of the current frame in consideration of the reused pixel value (page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration; page 50 col. 1 paragraph 6 – page 50 col. 2 paragraph 1: The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH). The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA). The TTH selects only one RGB-D pixel closest to the camera origin using a depth comparator during intra-tile handling. In the inter-tile handling, it skips the depth comparison but, by a bitwise ‘AND’ operation, determines overlapping pixels that need a new INF. The TFU reduces EMA by 99.5% and enables the VPC to reuse RGB-D pixels of the previous frame, resulting in 48.6× speed-up, on average … the binary map (Bmap) received from the TFU); and
determining whether additional sampling is necessary based on a result of the DNN inference (page 50 col. 1 paragraph 3: The TF stage first transforms RGB-D pixels of the previous frame into RGB-D pixels of the current frame, and then, with partial INF, evaluates the familiarity of the two frames by measuring color differences before and after the transformation. A small color difference implies high familiarity, allowing RGB-D pixels of the previous frame to be reused in the current frame rendering. Conversely, if familiarity is low, new RGB-D is computed through a new INF. In this way, more than 95% of DNN INFs can be skipped with <1 dB PSNR loss, compared to 9.7dB loss shown in naive RGB-D reuse without familiarity consideration. The TDA stage skips the last few layers of the DNN when the density obtained from the prior layers is low (short-term feedback). It also skips computing the remaining samples along a ray when the cumulative density of prior samples is high enough (long-term feedback). This reduces the total number of samples by 95.7% without additional PSNR loss; Han - page 50 col. 1 paragraph 6 – page 50 col. 2 paragraph 1: The TFU consists of a 3D transformation unit, O/O buffers and a tile-wise TF handler (TTH). The TFU projects the previous RGB-D pixels into the current frame via 3D transformation and receives results of partial lNF (0.72% of full INF) from the HNE to evaluate whether pixel data can be reused. Since the pixels within a tile show data locality even after 3D transformation, the TFU adopts relative addressing-based buffering to reduce 97.9% of the external memory accesses (EMA). The TTH selects only one RGB-D pixel closest to the camera origin using a depth comparator during intra-tile handling. In the inter-tile handling, it skips the depth comparison but, by a bitwise ‘AND’ operation, determines overlapping pixels that need a new INF. The TFU reduces EMA by 99.5% and enables the VPC to reuse RGB-D pixels of the previous frame, resulting in 48.6× speed-up, on average … the binary map (Bmap) received from the TFU).
Han does not explicitly teach the previous frame and a current frame corresponds to the observer changing positions; and expressing the object using low-resolution voxels generated in advance; then exclusively generating coordinates located inside the voxels in a gaze direction of the observer as sample coordinates subjected to DNN inference; and determining whether to continue inference based on a result of the DNN inference.
Han2 teaches the previous frame and a current frame corresponds to the observer changing positions (fig. 7 shows moving the camera viewpoints changes the position of the observer and generates a first frame from a first position and a next frame from a second position); and expressing the object using low-resolution voxels generated in advance (page 67 col. 1 last paragraph – page 67 col. 2 first paragraph: the proposed SA stage successfully reduces the number of samples while maintaining the rendering quality. Before it starts the rendering, it predefined low-resolution voxel to represent a 3-D model. This voxel can be obtained during the pretraining phase by indicating the sub-spaces which contain the high-density samples. After voxel acquisition, it follows the sampling method based on Fig. 4(b) but skips sampling outside the attention map. In other words, it collects only the meaningful samples inside the low-resolution voxel); then exclusively generate coordinates located inside the voxels in a gaze direction of the observer as sample coordinates subjected to DNN inference (shooting a ray and determining the samples lying on the same ray is functionally analogous to samples located inside the voxels in a gaze direction of the observer; page 70 col. 1 paragraph 2: The SAU consists of a sample coordinate generator (SCG), a 4-kB voxel cache, and a task controller. Since the TFU selects only a portion of pixels that need new RGB-D values, the role of SAU is to generate sample coordinates of the corresponding pixels. During the sample coordinate generation, the SAU shoots a ray based on the accumulated Bmap received from the TFU. It finds out the samples lying on the same ray while judging whether the samples are placed within the attention map; page 70 col. 1 paragraph 2: The SAU generates coordinates of the 8 samples per pixel by evaluating whether the sample is inside the attention map or not. Then, a tile is classified as dense or sparse tile depending on the number of valid samples per tile); and determining whether to continue inference based on a result of the DNN inference (page 68 col. 1 paragraph 2: The last stage, TDA, receives intermediate or final DNN INF results as feedback to determine whether it continues or skips the remaining operations. In the short-term feedback, it calculates density using the prior DNN layers. And then if the obtained density is low enough, it skips the calculation of the last few layers because it means that the sample is transparent. Although short-term feedback utilizes the intermediate density value, long-term feedback utilizes accumulated density as feedback. If the cumulative density is high enough, the results of the remaining samples do not affect the final result. The long-term feedback helps the SA stage and skips the computing of the remaining samples along a ray when the cumulated density of prior samples is high enough). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Han2’s knowledge of using the functionality of a VPC with a temporal familiarity unit and a spatial attention unit as taught and modify the process of Han because such a process provides a low-power neural 3D rendering processor for metaverse realization on mobile devices (abstract) that resolves the high computational cost problem and avoids performance degradation (page 72 col. 2 paragraph VI. Modulo-base PEU).
Regarding claim 13, the combination of Han and Han2 teaches the 3D rendering method according to claim 12, wherein the expressing the object comprises:
storing information about the voxels collected from an external memory in a cache memory (Han2 – page 70 col. 1 paragraph 2: The loading of the attention map is done by the built-in voxel cache to minimize EMA; it should be noted that EMA stands for external memory access, Han2 – page 69 col. 1 paragraph 1);
exclusively generating coordinates inside the voxels corresponding to the DNN inference target pixels among coordinates on a ray as sample coordinates by emitting the ray in the gaze direction of the observer (Han – page 50 col. 2 paragraph 1: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input; Han2 - page 69 col. 1 last paragraph: After 3-D transformation, multiple RGB-D pixels can be located on the same ray. This overlapping problem occurs within a tile (Intra-tile) or among the tiles (Inter-tile). Although the intra-tile overlapping can be discovered in a single calculation phase, the inter-tile overlapping can be found after a few calculation phases. Therefore, the built-in TTH deals with this problem by adopting two different ways. During the intra-tile handling, it selects only one RGB-D pixel closest to the camera origin using a depth comparator. On the contrary, the TTH skips depth comparison in the inter-tile handling. Instead, it finds out the overlapping pixels using the bitwise “AND” operation. The Bmap generated by the “AND” operation responds to the transformation results of adjacent tiles and the final accumulated Bmap indicates whether it needs an INF to get a new RGB-D value; Han2 – page 70 col. 1 paragraph 2: the role of SAU is to generate sample coordinates of the corresponding pixels. During the sample coordinate generation, the SAU shoots a ray based on the accumulated Bmap received from the TFU. It finds out the samples lying on the same ray while judging whether the samples are placed within the attention map. It can stop the calculation of the sample coordinates by receiving the accumulated density from TDA logic. Thanks to the help of TDA logic, the SAU can avoid the sample-coordinate generation of useless samples. The loading of the attention map is done by the built-in voxel cache to minimize EMA. It fetches only the new voxels using the cache misses, which minimizes the repeated voxel loading); and
classifying the tile-unit images into a dense tile or a sparse tile, and then controlling input/output (input controller and output controller) to/from an HNE according to a result of the classification (Han – page 50 col. 2 first paragraph: The SAU, which consists of the Sample Coordinate Generator (SCG), Voxel Cache, and Task Controller, generates sample coordinates using the binary map (Bmap) received from the TFU and accumulated density of the TDA logic as an input. The Voxel Cache uses the coordinates to fetch new voxels only in the case of cache misses. The voxel is used to evaluate whether the sample is inside the attention map or not, and then the tile is transferred to the either the dense or sparse buffer of the Task Controller, depending on the number of valid samples per tile. The Task Controller manages the I/O data for the HNE with out-of-order execution (OoOE). The input controller transfers dense tiles directly to the HNE, but in the case of sparse tiles, it waits until input coordinates are fully accumulated in a buffer. The output controller decodes the HNE output with either the Masking Unit or Reorder Buffer (ROB) to generate final RGB-D results.; Han2 – page 70 col. 1 last paragraph: Since the efficiency of the HNE can be further improved when enough samples exist in a tile, the task controller manages the I/O data of the HNE with out-of-order execution (OoOE) as described in Fig. 9. The input controller transfers dense tiles directly to the HNE, but in the case of sparse tiles, it waits until input coordinates are accumulated enough in a buffer. The output controller decodes HNE output using either the masking unit or the reorder buffer (ROB) to generate final RGB-D results).
Claims 14, 15, 16, 17, 18 , 19 and 20 are similar in scope to claims 3, 4, 5, 7, 10, 8 and 9, respectively, and therefore, the examiner provides similar rationale to reject these claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Han et al. (“A Low-Power Artificial-Intelligence-Based 3-D Rendering Processor With Hybrid Deep Neural Network Computing”) describes a low-power artificial intelligence (AI)-based 3-D rendering processor is proposed for metaverse solutions in mobile platforms. It suggests a brain-inspired rendering acceleration architecture designed with a visual perception core. It removes useless computations by realizing 1) spatial attention, 2) temporal familiarity, and 3) top-down attention. The remaining deep neural network (DNN) inference tasks are accelerated by a hybrid neural engine that utilizes both coarse-grained and fine-grained sparsity exploitation simultaneously. It divides the DNN tasks into sparse and dense data and allocates them to the two different neural engines, which focus on zero skipping and data reusability, respectively. Thanks to the centrifugal sampling-based workload prediction, it can dynamically divide DNN computations while minimizing peak signal-to noise ratio loss caused by the prediction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JWALANT B AMIN whose telephone number is (571)272-2455. The examiner can normally be reached Monday-Friday 10am - 630pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JWALANT AMIN/Primary Examiner, Art Unit 2612
Prosecution Insights