DETAILED ACTION
This action is in response to the application filed on 12/16/2024.
Claims 1-20 are pending.
Acknowledgment is made of a claim for foreign priority. None of the certified copies of the priority documents have been received.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The references listed on the Information Disclosure Statement submitted on 12/16/2024 has/have been considered by the examiner (see attached PTO-1449).
Claim Mapping Notation
In this office action, following notations are being used to refer to the paragraph numbers or column number and lines of portions of the cited reference.
In this office action, following notations are being used to refer to the paragraph numbers or column number and lines of portions of the cited reference.
[0005] (Paragraph number [0005])
C5 (Column 5)
Pa5 (Page 5)
S5 (Section 5)
Furthermore, unless necessary to distinguish from other references in this action, “et al.” will be omitted when referring to the reference.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 6-14, 16-18 and 20 are rejected under 35 U.S.C. 102(a2) as being anticipated by Xu et al. (US 20260170700 A1)
1. An image processing network module comprising:
an encoder configured to receive input image data and change a bit depth of the input image data to generate first image data;
“[0029] To resolve these issues, a method and system disclosed herein uses an input pre-processing neural network (PRPNN) at an encoder side that converts an input frame into one or more image channels that are a representation of the image with less bits.”
a quantization network configured to quantize the first image data to generate second image data; and
“[0098] The pre-processing unit 1506 may have an input PRP NN 1516 with an input bit-depth unit 1518 either as part of the PRP NN 1506 or communicating therewith. The video encoder unit 1512 may have encoder components such as partitioning units, transform, such as (DCT) and quantization units, decoder loop units, prediction units, including a motion estimation unit that uses motion vectors, and residual generating units, entropy unit, and so forth.”
a decoder configured to receive the second image data and change a bit depth of the second image data to generate output image data.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
2. The image processing network module of claim 1, wherein the input image data comprises m-bit data, wherein each of the first image data and the second image data comprises n-bit data, and wherein m and n are positive integers and n is less than m.
“[0029] To resolve these issues, a method and system disclosed herein uses an input pre-processing neural network (PRPNN) at an encoder side that converts an input frame into one or more image channels that are a representation of the image with less bits.”
3. The image processing network module of claim 1, wherein the output image data is m-bit data or n-bit data, and wherein m and n are positive integers and n is less than m.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
4. The image processing network module of claim 1, wherein the encoder is configured to truncate at least one bit of the input image data to change the bit depth of the input image data.
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
6. The image processing network module of claim 4, wherein the encoder is configured to truncate at least two bits of the input image data.
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
7. The image processing network module of claim 1, wherein the decoder is configured to change the bit depth of the second image data by restoring at least one bit of the second image data.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
8. The image processing network module of claim 7, wherein the decoder is configured to restore the at least one bit of the second image data from a least significant bit (LSB) of the second image data to a most significant bit (MSB) of the second image data.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
9. The image processing network module of claim 1, wherein the image processing network module is configured to perform image processing on the input image data by using a neural network model trained to perform preset image processing operations to generate the output image data.
“[0028] Other example systems with two neural networks merely use a pre- and post-processing network that downscales and then performs video super-resolution or upscaling operations to recover texture details. Not only are these systems not true dual neural network systems that learn different image domains that can be compressed and decompressed as mentioned above, these systems are often inadequate because these systems cannot regain enough detail lost in the down-scaled inputs.”
10. The image processing network module of claim 9, wherein the neural network model is trained to perform at least one of a remosaic operation, a super-resolution operation, and a deblurring operation.
“[0028] Other example systems with two neural networks merely use a pre- and post-processing network that downscales and then performs video super-resolution or upscaling operations to recover texture details. Not only are these systems not true dual neural network systems that learn different image domains that can be compressed and decompressed as mentioned above, these systems are often inadequate because these systems cannot regain enough detail lost in the down-scaled inputs.”
11. An image processing device comprising:
a camera configured to receive an input image; and
“[0096] Referring to FIG. 15, an example video coding system or device 1500 for providing video coding with pre-processing and post-processing neural networks may be arranged in accordance with at least some implementations of the present disclosure. In the illustrated implementation, system 1500 may include imaging device(s) 1501 such as one or more cameras,”
a neural network processor configured to generate input image data by dividing the input image into blocks of a predetermined size and configured to generate output image data by performing image processing on the input image data by using a neural network model trained to perform preset image processing operations,
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
“[0098] The pre-processing unit 1506 may have an input PRP NN 1516 with an input bit-depth unit 1518 either as part of the PRP NN 1506 or communicating therewith. The video encoder unit 1512 may have encoder components such as partitioning units, transform, such as (DCT) and quantization units, decoder loop units, prediction units, including a motion estimation unit that uses motion vectors, and residual generating units, entropy unit, and so forth.”
“[0028] Other example systems with two neural networks merely use a pre- and post-processing network that downscales and then performs video super-resolution or upscaling operations to recover texture details. Not only are these systems not true dual neural network systems that learn different image domains that can be compressed and decompressed as mentioned above, these systems are often inadequate because these systems cannot regain enough detail lost in the down-scaled inputs.”
wherein the neural network processor comprises:
an encoder configured to receive the input image data and change a bit depth of the input image data to generate first image data;
“[0029] To resolve these issues, a method and system disclosed herein uses an input pre-processing neural network (PRPNN) at an encoder side that converts an input frame into one or more image channels that are a representation of the image with less bits.”
a quantization network configured to quantize the first image data to generate second image data; and
“[0098] The pre-processing unit 1506 may have an input PRP NN 1516 with an input bit-depth unit 1518 either as part of the PRP NN 1506 or communicating therewith. The video encoder unit 1512 may have encoder components such as partitioning units, transform, such as (DCT) and quantization units, decoder loop units, prediction units, including a motion estimation unit that uses motion vectors, and residual generating units, entropy unit, and so forth.”
a decoder configured to receive the second image data and change a bit depth of the second image data to generate the output image data.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
12. The image processing device of claim 11, wherein the neural network model is trained to perform at least one of a remosaic operation, a super-resolution operation, and a deblurring operation.
“[0028] Other example systems with two neural networks merely use a pre- and post-processing network that downscales and then performs video super-resolution or upscaling operations to recover texture details. Not only are these systems not true dual neural network systems that learn different image domains that can be compressed and decompressed as mentioned above, these systems are often inadequate because these systems cannot regain enough detail lost in the down-scaled inputs.”
13. The image processing device of claim 11, wherein the input image data is m-bit data, wherein each of the first image data and the second image data is n-bit data, and wherein n and m are positive integers, and n is less than m.
“[0029] To resolve these issues, a method and system disclosed herein uses an input pre-processing neural network (PRPNN) at an encoder side that converts an input frame into one or more image channels that are a representation of the image with less bits.”
14. The image processing device of claim 11, wherein the encoder is configured to truncate at least one bit of the input image data to change the bit depth of the input image data, and wherein the decoder is configured to restore at least one bit of the second image data to change the bit depth of the second image data.
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
16. The image processing device of claim 14, wherein the decoder is configured to generate the output image data by restoring a least significant bit (LSB) of the second image data.
“[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
17. The image processing device of claim 11, further comprising a display configured to display an image based on the output image data.
“[0096] Referring to FIG. 15, an example video coding system or device 1500 for providing video coding with pre-processing and post-processing neural networks may be arranged in accordance with at least some implementations of the present disclosure. In the illustrated implementation, system 1500 may include imaging device(s) 1501 such as one or more cameras, one or more central and/or graphics processing units or processors 1503, a display device 1505, one or more memory stores 1507, an antenna 1550 for wireless transmission, and processing unit(s) 1502 to perform the operations mentioned above.”
Regarding claims 18 and 20 they recite elements that are at least included in the claims 11 and 13 but in a different claim form. Therefore, the same rationale for the rejection of claims 11 and 13 applies.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 5, 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Yazawa et al. (US 5677860 A).
Regarding claim 5, Xu discloses the invention substantially as claimed as mentioned above for claims 1 and 4.
Xu discloses,
5. The image processing network module of claim 4, wherein the encoder is configured to…
…and truncate the at least one bit of the input image data…
“[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
Xu does not disclose,
…compare a first most significant bit (MSB) and a second MSB of the input image data with a preset maximum value… truncate the at least one bit of the input image data based on a result of comparison.
Yazawa discloses,
…compare a first most significant bit (MSB) and a second MSB of the input image data with a preset maximum value… truncate the at least one bit of the input image data based on a result of comparison.
C3 “According to another aspect of the present invention, a processing circuit is provided for adding a plurality of data of N bits with an adder to limit the number of digits to N-m bits for processing an overflow/underflow. A plurality of signals indicating presence of overflow/underflow are generated in advance with a signal including the more significant bits of a plurality of data of N bits. One of the plurality thereof is selected using a signal including a carry-out signal or a sum signal of the adder.”
C9 “FIG. 15 shows a calculation where 9-bit two's complements are added with other for providing a two's complement in the range of 8 bits. The output of the present embodiment is 8 bits. In order to carry out an overflow/underflow process with adders of 8 bits, a determination of the presence of an overflow/underflow is made by calculation for all the combinations of horizontal direction most significant bit H8, the second most significant bit H7, vertical direction most significant bit V8, the second most significant bit V7, and the carry-in signal to the adder of the seventh bit.”
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to utilize the teachings of Yazawa and apply them on the teachings of Xu to incorporate the bit reduction scheme of Yazawa in Xu as taught by Yazawa.
One would have been motivated for the benefit of insuring overly drastic change of the bit depth which may degrade the image quality significantly as taught by Yazawa.
Unless stated otherwise, the same explanation for the rationale for the following dependent claims applies as given for the independent claim.
15. The image processing device of claim 14, wherein a first most significant bit (MSB) and a second MSB of the input image data are compared with a preset maximum value and truncate at least two bits of the input image data.
Xu “[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
C3 “According to another aspect of the present invention, a processing circuit is provided for adding a plurality of data of N bits with an adder to limit the number of digits to N-m bits for processing an overflow/underflow. A plurality of signals indicating presence of overflow/underflow are generated in advance with a signal including the more significant bits of a plurality of data of N bits. One of the plurality thereof is selected using a signal including a carry-out signal or a sum signal of the adder.”
C9 “FIG. 15 shows a calculation where 9-bit two's complements are added with other for providing a two's complement in the range of 8 bits. The output of the present embodiment is 8 bits. In order to carry out an overflow/underflow process with adders of 8 bits, a determination of the presence of an overflow/underflow is made by calculation for all the combinations of horizontal direction most significant bit H8, the second most significant bit H7, vertical direction most significant bit V8, the second most significant bit V7, and the carry-in signal to the adder of the seventh bit.”
19. The method of claim 18, wherein the generating the first image data comprises comparing a first most significant bit (MSB) and a second MSB of the input image data with a preset maximum value to truncate at least two bits of the input image data, and
Xu “[0149] In example 20, the subject matter of example 18 or 19 wherein one or consecutive multiple least significant bit (LSB) locations are dropped to reduce bit-depth of the encoder-input image data relative to the initial image data.”
Yazawa C3 “According to another aspect of the present invention, a processing circuit is provided for adding a plurality of data of N bits with an adder to limit the number of digits to N-m bits for processing an overflow/underflow. A plurality of signals indicating presence of overflow/underflow are generated in advance with a signal including the more significant bits of a plurality of data of N bits. One of the plurality thereof is selected using a signal including a carry-out signal or a sum signal of the adder.”
Yazawa C9 “FIG. 15 shows a calculation where 9-bit two's complements are added with other for providing a two's complement in the range of 8 bits. The output of the present embodiment is 8 bits. In order to carry out an overflow/underflow process with adders of 8 bits, a determination of the presence of an overflow/underflow is made by calculation for all the combinations of horizontal direction most significant bit H8, the second most significant bit H7, vertical direction most significant bit V8, the second most significant bit V7, and the carry-in signal to the adder of the seventh bit.”
wherein the generating the output image data comprises restoring an LSB of the second image data.
Xu “[0037] The output-decoder image data 124 is provided to the post-processing unit 126 and input into the POPNN 128 to generate low distortion output images 132 that are similar or the same as input images 102, and with much lower distortion than coding without the neural networks, and better performance than without the bit-depth reduction. The bit-depth adjuster 130 of the post-processing unit 126, as with the bit-depth adjuster 110, may be considered part of the POPNN 128 or separate from the network. The bit-depth adjuster 130 is arranged to increase the bit-depth of the bit-depth-reduced channels of the output-decoder images 124, usually back to the original bit-depth of the input images 102, but need not always be so.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Shi et al. (US 20170270408 A1) discloses relevant art related to the subject matter of the present invention.
A shortened statutory period for reply to this action is set to expire THREE MONTHS from the mailing date of this action. An extension of time may be obtained under 37 CFR 1.136(a). However, in no event, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAE N NOH whose telephone number is (571)270-0686. The examiner can normally be reached on Mon-Fri 8:30AM-5PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached on (571) 272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAE N NOH/
Primary Examiner
Art Unit 2481
