DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 7 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites “a time warping circuit implemented as a hardened circuit block, wherein the time warping circuit is configured to operate concurrently and independently of a low latency data path of the ISP circuit.” It is unclear what is the low latency data path of the IPS circuit. Low latency is a relative phrase. How to define what is low latency and what is the reference. Without clear scope of this phrase, it would have no way to specify the claim scope of “concurrently and independently of…” Clarification is required. For compact prosecution, in the prior art rejection below, Examiner will provide rejection based on the limitation of “a time warping circuit implemented as a hardened circuit block”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-10, 13-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US 2023/0070191 A1) in view of Ogale (US 8666159 B1).
Regarding claim 1, Kim teaches:
A system, comprising: a Visual See-Through (VST) pipeline circuit capable of processing image data, (Abstract: “A system on chip and a mobile device are provided. The mobile device comprises a processor configured to receive raw image data, process the raw image data into floating-point format image data, and output the floating-point format image data, a memory configured to store therein the floating-point format image data, and a display processing unit configured to receive the floating-point format image data stored in the memory therefrom, and perform high dynamic range (HDR) processing on the floating-point format image data.”)
wherein the VST pipeline circuit is embodied as a die ([0032], “an electronic device 1 may include a system on chip (hereinafter referred to as a SoC) 10”) and includes:
an Image Signal Processing (ISP) circuit; ([0042], “Referring to FIG. 4, the SoC 10 may include an ISP (Image Signal Processor) 110,”) a Display Processing Unit (DPU) circuit coupled to the ISP circuit; (FIG. 4, 120 display processing unit) and a memory circuit coupled to the ISP circuit and to the DPU circuit, (FIG. ) wherein the memory circuit is configured to implement a plurality of buffers that facilitate low latency operation of the ISP circuit and the DPU circuit.(FIG. 4, 200 and 30, [0124] teaches internal memory. Internal memory will lower image transferring speed comparing to external memory: “The application processor 1200 may store the received image signal, that is, the encoded image signal in an internal memory 1230 or storage 1400 external to the application processor 1200, and then may read out the encoded image signal from the memory 1230 or the storage 1400 and then may decode the read-out signal and may display image data generated based on the decoded image signal.”)
However, Kim does not teach:
the memory circuit is on the die
On the other hand, Ogale teaches:
the memory circuit is on the die (para 12: “he GPUs and/or CPUs of the mobile device may comprise multiple memory portions. For example, the GPU may comprise one or more on-chip memory portions including one or more "textures" and/or one or more "frame buffers." Each of the on-chip memory portions may be logically divided into one or more discrete textures (e.g., for reading image data into the GPU) and/or one or more discrete frame buffers (e.g., for storing processed image data outputted from the GPU).”)
Kim teaches a system on chip (SoC) where ISP, DPU and memory control are all embodied on the chip. However, Kim does not teach that the memory can be on the chip. On the other hand, Ogale teaches a memory can be on-chip memory where the memory buffer are on the same chip with the image processing circuit to improve image data transferring time.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kim with the specific teachings of Ogale to include the memory buffers on the same chip of the processing circuit to improve image data transferring efficiency.
Regarding claim 2, Kim in view of Ogale teaches:
The system of claim 1, wherein the ISP circuit is configured to process image data received from one or more cameras; (Kim [0047], “For example, the ISP 110 may convert first image data having a first format output from the camera modules 71, 72, and 73 into second image data having a second format.”) and wherein the DPU circuit is configured to generate blended frames by blending image data output from the ISP circuit with image data specifying one or more digital content items generated by a Graphics Processing Unit (GPU).( Kim [0115], “In some example embodiments, the plurality of sub-image processors 1212a, 1212b, and 1212c may correspond to the ISP (110 in FIG. 4) as described above.” Kim teaches DPU can combine data received from different processors and the processor can be ISP circuit. On the other hand, Kim teaches the DPU will get image data form memory 30 which are processed by the GPU. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the two parts of teachings of Kim to allow DPU to combine images from both ISP and GPU to enable the system to provide a more flexible image generation capability.)
Regarding claim 3, Kim in view of Ogale teaches:
The system of claim 2, wherein the DPU circuit is configured to output the blended frames to a display device.( Kim [0115], “Specifically, the image generator 1214 may merge at least some of the image data respectively generated from the image processors 1212a, 1212b, and 1212c based on the image generation information or the mode signal, thereby generating the output image as the merging result.” [0114], “The image generator 1214 may generate an output image using the image data provided from each of the sub-image processors 1212a, 1212b, and 1212c based on image generation information or a mode signal. In some example embodiments, the image generator 1214 may correspond to the DPU (400 in FIG. 4) as described above.”)
Regarding claim 4, Kim in view of Ogale teaches:
The system of claim 1, wherein the memory circuit is implemented as a static random-access memory or an embedded dynamic random-access memory. (Kim, [0040], “The memory 30 may be used as, for example, a main memory for the electronic device 1, and may include a volatile memory such as SRAM or DRAM.”)
Regarding claim 5, Kim in view of Ogale teaches:
The system of claim 1, wherein the ISP circuit comprises a plurality of hardened circuit blocks configured to perform image processing operations coupled by one or more of the plurality of buffers. (Kim, [0046], “The ISP 110 may convert a format of raw image data output from the camera modules 71, 72, and 73, or perform operations for noise reduction, and image enhancement on the raw image data.”[0108], “In some example embodiments, the plurality of sub-image processors 1212a, 1212b, and 1212c may correspond to the ISP (110 in FIG. 4) as described above.” [0128], Ogale :para 12: “he GPUs and/or CPUs of the mobile device may comprise multiple memory portions. For example, the GPU may comprise one or more on-chip memory portions including one or more "textures" and/or one or more "frame buffers." Each of the on-chip memory portions may be logically divided into one or more discrete textures (e.g., for reading image data into the GPU) and/or one or more discrete frame buffers (e.g., for storing processed image data outputted from the GPU).” The combination of claim 1 is incorporated here.)
Regarding claim 6, Kim in view of Ogale teaches:
The system of claim 5, wherein the plurality of hardened circuit blocks are coupled by a multiplexer circuit capable of bypassing one or more selected hardened circuit blocks of the plurality of hardened circuit blocks responsive to control signals.(Kim, [0111], “Further, in some example embodiments, the image data generated from the camera module 1100a may be provided to the sub-image processor 1212a via the image signal line ISLa. The image data generated from the camera module 1100b may be provided to the sub-image processor 1212b via the image signal line ISLb. The image data generated from the camera module 1100c may be provided to the sub-image processor 1212c via the image signal line ISLc. Further, the image data processed by the sub-image processor 1212b may be provided directly to the image generator 1214. One of the image data processed by the sub-image processor 1212a and the image data processed by the sub-image processor 1212c may be selected by a selection element (e.g., a multiplexer). The selected one may be provided to the image generator 1214.”)
Regarding claim 8, Kim in view of Ogale teaches:
The system of claim 1, wherein the DPU circuit comprises a plurality of hardened circuit blocks coupled by one or more of the plurality of buffers and configured to perform image processing operations. (kim, [0066], “Referring to FIG. 7, the DPU 400 may include an interfacing device 410, a first converter 420, an integer image processing unit 430, a second converter 440, a first multiplexer 450, a floating-point image processing unit 460, and a second multiplexer 470.”)
Regarding claim 9, Kim in view of Ogale teaches:
The system of claim 8, wherein the plurality of hardened circuit blocks are coupled by a multiplexer circuit capable of bypassing one or more selected hardened circuit blocks of the plurality of hardened circuit blocks responsive to control signals. (Kim [0069], “The first converter 420 may be enabled in response to a predefined enable signal. For example, when the image data provided from the memory 30 is the floating-point format image data FPID, the first converter 420 may be enabled to perform a conversion operation so that integer arithmetic may be performed on the floating-point format image data FPID by the integer image processing unit 430. To the contrary, when the image data provided from the memory 30 is the floating-point format image data FPID, the first converter 420 may be disabled, because and the integer arithmetic is not required.”)
Regarding claim 10, Kim in view of Ogale teaches:
The system of claim 8, wherein the plurality of hardened circuit blocks of the DPU circuit implement, at least in part: a first blending channel capable of processing image data obtained from a camera;( Kim [0111], “Further, in some example embodiments, the image data generated from the camera module 1100a may be provided to the sub-image processor 1212a via the image signal line ISLa. The image data generated from the camera module 1100b may be provided to the sub-image processor 1212b via the image signal line ISLb. The image data generated from the camera module 1100c may be provided to the sub-image processor 1212c via the image signal line ISLc. Further, the image data processed by the sub-image processor 1212b may be provided directly to the image generator 1214. One of the image data processed by the sub-image processor 1212a and the image data processed by the sub-image processor 1212c may be selected by a selection element (e.g., a multiplexer). The selected one may be provided to the image generator 1214.”) a second blending channel capable of processing image data generated by a Graphics Processing Unit; (Kim [0053], “The GPU 120 may perform rendering on the raw image data provided, for example, from the memory 30. The thus rendered image data may be transferred to the DPU 400 via the memory 30.”) and a blending circuit capable of blending image data output from the first blending channel with image data output from the second blending channel. (Kim [0115], “he image generator 1214 may merge at least some of the image data respectively generated from the image processors 1212a, 1212b, and 1212c based on the image generation information or the mode signal, thereby generating the output image as the merging result. Further, the image generator 1214 may select one of the image data respectively generated from the image processors 1212a, 1212b, and 1212c based on the image generation information or the mode signal, thereby generating the output image as the selected one..” Kim teaches DPU can combine data received from different processors and the processor can be ISP circuit. On the other hand, Kim teaches the DPU will get image data form memory 30 which are processed by the GPU. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the two parts of teachings of Kim to allow DPU to combine images from both ISP and GPU to enable the system to provide a more flexible image generation capability.)
Regarding claim 13, Kim in view of Ogale teaches:
The system of claim 1, further comprising: a Central Processing Unit (CPU) configured to control operation of the DPU circuit and the ISP circuit; (Kim, [0045], “The CPU 300 may control the ISP 110, the GPU 120, the MFC 130, the memory controller 200, and the DPU 400 via the bus 500.”) and a Graphics Processing Unit (GPU) configured to generate one or more digital content items to be overlayed on image data output from a camera. (Kim [0115], “the image generator 1214 may merge at least some of the image data respectively generated from the image processors 1212a, 1212b, and 1212c based on the image generation information or the mode signal, thereby generating the output image as the merging result. Further, the image generator 1214 may select one of the image data respectively generated from the image processors 1212a, 1212b, and 1212c based on the image generation information or the mode signal, thereby generating the output image as the selected one..” Kim teaches DPU can combine data received from different processors and the processor can be ISP circuit. On the other hand, Kim teaches the DPU will get image data form memory 30 which are processed by the GPU. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the two parts of teachings of Kim to allow DPU to combine images from both ISP and GPU to enable the system to generated combined images.)
Regarding claim 14, Kim in view of Ogale teaches:
The system of claim 13, wherein the CPU is embodied in the die with the ISP circuit, the DPU circuit, and the memory circuit.(Kim, See FIG. 4, the CPU is on the SoC)
Regarding claim 15, Kim in view of Ogale teaches:
The system of claim 13, wherein the GPU is embodied in the die with the ISP circuit, the DPU circuit, and the memory circuit. (Kim, See FIG. 4, the GPU is on the SoC)
Regarding claim 16, Kim in view of Ogale teaches:
The system of claim 13, wherein the CPU and the GPU are embodied in the die with the ISP circuit, the DPU circuit, and the memory circuit. (Kim, See FIG. 4, the GPU and CPU are on the SoC)
Regarding claim 17, Kim teaches:
A method, comprising:
processing image data through a first portion of a Visual See-Through (VST) pipeline circuit including an Image Signal Processor (ISP) circuit having a first plurality of hardened circuit blocks; (See FIG. 4, SoC, the ISP 110 is on the SoC, [0046], “The ISP 110 may convert a format of raw image data output from the camera modules 71, 72, and 73, or perform operations for noise reduction, and image enhancement on the raw image data.”[0108], “In some example embodiments, the plurality of sub-image processors 1212a, 1212b, and 1212c may correspond to the ISP (110 in FIG. 4) as described above.” [0128]) wherein the first plurality of hardened circuit blocks of the ISP circuit are coupled by …memory; ([0050], “An output of the ISP 110 may be sent to the DPU 400 via the memory 30.”) and
processing image data output from the ISP circuit through a second portion of the VST pipeline circuit including a Display Processing Unit (DPU) circuit having a second plurality of hardened circuit blocks; (See FIG. 4, the DPU 120 is on the SoC. [0063], “The DPU 400 may receive the image data output from the ISP 110, the GPU 120, and the MFC 130, and may perform predefined image processing thereon.” [0066], “Referring to FIG. 7, the DPU 400 may include an interfacing device 410, a first converter 420, an integer image processing unit 430, a second converter 440, a first multiplexer 450, a floating-point image processing unit 460, and a second multiplexer 470.” [0128])
wherein the second plurality of hardened circuit blocks of the DPU circuit are coupled by … memory. ([0050], “An output of the ISP 110 may be sent to the DPU 400 via the memory 30.”)
However, Kim does not teach:
ISP circuits are coupled by a first plurality of buffers of an on-die the memory
DPU circuit are coupled by a second plurality of buffers of the on-die memory.
On the other hand, Ogale teaches:
a processor is coupled by a plurality of buffers of an on-die the memory
(para 12: “he GPUs and/or CPUs of the mobile device may comprise multiple memory portions. For example, the GPU may comprise one or more on-chip memory portions including one or more "textures" and/or one or more "frame buffers." Each of the on-chip memory portions may be logically divided into one or more discrete textures (e.g., for reading image data into the GPU) and/or one or more discrete frame buffers (e.g., for storing processed image data outputted from the GPU).”)
Kim teaches a system on chip (SoC) where ISP, DPU and memory control are all embodied on the same chip. However, Kim does not teach the memories coupled by ISP and DPU can be on the same chip. On the other hand, Ogale teaches a memory can be on-chip memory where the memory buffers are on the same chip with the image processing circuit to improve image data transferring time.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kim with the specific teachings of Ogale to include the memory buffers on the same chip of the processing circuit to improve image data transferring efficiency.
Regarding claim 18, Kim in view of Ogale teaches:
The method of claim 17, wherein the ISP circuit, the DPU circuit, and the on-die memory are implemented on a single die. (Kim, See FIG. 4, ISP and DPU are on the same SoC. For on-die memory teachings, please refer to Claim 17 teachings and combination.)
Claim 19 recites similar limitations of claim 2, thus are rejected accordingly.
Regarding claim 20, Kim in view of Ogale teaches:
The method of claim 17, wherein the plurality of hardened circuit blocks of at least one of the ISP circuit or the DPU circuit are coupled by a multiplexer circuit, the method further comprising: bypassing one or more selected hardened circuit blocks of the plurality of hardened circuit blocks responsive to control signals provided to the multiplexer circuit. (Kim [0069], “The first converter 420 may be enabled in response to a predefined enable signal. For example, when the image data provided from the memory 30 is the floating-point format image data FPID, the first converter 420 may be enabled to perform a conversion operation so that integer arithmetic may be performed on the floating-point format image data FPID by the integer image processing unit 430. To the contrary, when the image data provided from the memory 30 is the floating-point format image data FPID, the first converter 420 may be disabled, because and the integer arithmetic is not required.”)
Claim(s) 7, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Ogale and further in view of Brailovskiy et al. (US 2024/0338073 A1).
Regarding claim 7, Kim in view of Ogale teaches:
The system of claim 5, further comprising:
However, Kim in view of Ogale does not, but Brailovskiy teaches:
a time warping circuit implemented as a hardened circuit block, wherein the time warping circuit is configured to operate concurrently and independently of a low latency data path of the ISP circuit. ([0054], “The output of the image sensor is sent to ISP processing 516 as described herein. The ISP processing 516 may also use the gaze-driven foveation map 512 to perform ISP processing 516. The processed image is sent to the GPU 508 to go through warping and fusion 518. In particular embodiments, the GPU 508 may receive other inputs 520, such as depth measurements and VR content. In particular embodiments, the GPU 508 may use the gaze-driven foveation map 512 for one or more processes. After the processed image goes through the warping and fusion process 518, the output goes through a composition and time warp process 522. The composition and time warp process 522 may use the gaze-driven foveation map 512 and one or more other inputs 520 to perform the composition and time warp.”)
Kim in view of Ogale teaches an ISP to do image processing. Brailovskiy further teaches to include a time warping circuit to further process images.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kim in view of Ogale with the specific teachings of Brailovskiy to enable the system to further process images as needed.
Regarding claim 11, Kim in view of Ogale teaches:
The system of claim 10, wherein each of the first blending channel and the second blending channel (see claim 10)
However, Kim in view of Ogale does not, but Brailovskiy teaches:
wherein blending channel includes one or more of: a foveated upscaling circuit; or a correction circuit capable of correcting one or more of distortion warp, shearing warp, or late-stage warp in image data.([0054], “After the processed image goes through the warping and fusion process 518, the output goes through a composition and time warp process 522. The composition and time warp process 522 may use the gaze-driven foveation map 512 and one or more other inputs 520 to perform the composition and time warp. The GPU 508 sends an output image to the display 510, which may be similar to the output sent to the display process 410 (shown in FIG. 4B). The display 510 performs an upscale and display process 524 on the image received from the GPU 508. The display 510 outputs images 526 to the displays of a computing system”)
Kim in view of Ogale teaches images blending channels. Brailovskiy teaches a blending channel includes a foveated upscaling circuit to generating better quality image for users to view.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kim in view of Ogale with the specific teachings of Brailovskiy to generate high quality blended images.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Ogale and further in view of Cai et al. (US 2020/0026087 A1).
Regarding claim 12, Kim in view of Ogale teaches:
The system of claim 8,
However, Kim in view of Ogale does not, but Cai teaches:
wherein the plurality of hardened circuit blocks include one or more of: a 3D color lookup table circuit, wherein the 3D color lookup table circuit is configured to store color enhancement data; a detail enhancement circuit configured to offset an effect of scaler related blurring; or an optical uniformity correction circuit configured to reduce brightness roll off. ([0068], “Optical region 1170 comprises a lensing surface near display panel 1120R such as a QPI display device and is provided with a positive optical power. Provided functions of optical region 1170 may include: 1) increasing the magnification of the optical system beyond that of lower optically reflective surface 1180; 2) balancing the Petzval field curvature of lower optically reflective surface 1180 for better image quality over a larger field; and; 3) to control the display panel 1120R light emission direction to reduce image brightness roll off.”)
Kim in view of Ogale teaches acquiring images and processing image and controlling the images to be presented on a display. Cai teaches a circuit to perform reducing image brightness roll off before present the images to a display.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kim in view of Ogale with the specific teachings of Cai to generate more high quality images to users.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/Primary Examiner, Art Unit 2615
Prosecution Insights