DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the claims dated 8/9/2024.
Claims 1-20 are presented for examination.
Priority
Acknowledgement is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been received for the foreign priority application no. KR 10-2023-0105148 filed 8/10/2023.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 8/9/2024, 2/3/2025, and 11/13/2025 have been considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: reference character 200, which appears twice in paragraph [0116] and there designates the display device, and reference character 1200, which appears in paragraph [0173] and there designates the display device. Neither character appears in any of the seventeen figures. Each appears to be an inadvertent rendering of reference character 2000, which the description uses throughout and which FIG. 1, FIG. 10 and FIG. 11 use to designate the display device. Accordingly, and in lieu of adding reference characters 200 and 1200 to the drawings, applicant is permitted to amend the description so that each of these three occurrences reads 2000, provided that no new matter is introduced. If applicant instead elects to add the reference characters to the drawings, corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to this Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. See 37 CFR 1.85(a).
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The present title, "Display Device and Operating Method Thereof," names only the field of the invention and conveys nothing of the subject matter to which claims 1, 11 and 20 are directed, namely obtaining a lightweight convolutional neural network having weights of reduced bit width, determining a shift distance based on a value of input feature data, reducing the bit width of that input feature data by a shift operation, and performing a convolution operation that itself includes a shift operation restoring the bit width.
The following title is suggested: "Display Device Performing a Convolution Operation With Dynamic Bit Width Reduction and Restoration Using a Lightweight Convolutional Neural Network, and Operating Method Thereof".
The disclosure is objected to because of the following informalities:
(a) paragraph [0042] sets forth the clause "or may be implemented through variations of various deep neural network architectures and algorithms" twice in immediate succession;
(b) paragraph [0067] recites "may determines a unit step S of a shift distance", which should read "may determine";
(c) paragraph [0115] and paragraph [0119] each refer to reference character 730 as a combined short-bit weight, although FIG. 7A shows 730 as the combined weight obtained from the first weight 710 and the second weight 720, and FIG. 7B shows 730 as the weight supplied to the dynamic shift operation 740 and shows 732 as the resulting combined short-bit weight, so that the later sentence of paragraph [0119], which attributes the information bits and the shift bits to reference character 730, appears to be directed to reference character 732;
(d) paragraph [0119] recites that the display device stores "the value of the feature data remaining after the shift operation" in the information bits, although that paragraph is directed to the combined weight rather than to feature data;
(e) paragraph [0132] contains an extraneous space before the comma in "In an embodiment , an original weight 910";
(f) paragraph [0134] and paragraph [0148] each recite "When a conditions of", which should read "When a condition of";
(g) paragraph [0138] recites "N16" and paragraph [0152] recites "N14", in each of which an equals sign appears to have been omitted, as shown by the corresponding sentence at paragraph [0144], which recites "N = 10"; and
(h) paragraph [0169] recites "but embodiments are limited thereto", which in view of the parallel statements at paragraphs [0032], [0033], [0042] and [0048] should read "but embodiments are not limited thereto".
Appropriate correction is required.
Claim Objections
Claim 10 is objected to because of the following informalities:
Claim 10 recites "The method of claim 9, the method further comprises determining", which is not a proper transitional form for a dependent claim and should read "The method of claim 9, further comprising determining".
Appropriate correction is required.
Claim Rejections - 35 U.S.C. 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 11 and 20 each recite the limitation "a shift operation of restoring the bit width" in the wherein clause that defines the convolution operation, and each recites the limitation "the restored bit width" in the final step. There is insufficient antecedent basis for these limitations in the claims. See MPEP 2173.05(e). Before the restoring limitation is reached, each of claims 1, 11 and 20 has recited three separate bit widths: "a reduced bit width of weights"; "a bit width of the input feature data," which the claim then reduces; and "the reduced bit width" that the input feature data and the weights possess when the convolution operation is performed. The recitation "the bit width" that is restored does not identify which of those three bit widths is the one restored. Where more than one candidate antecedent has been recited, a later definite reference is unclear because it is uncertain which of the recited elements was intended. See MPEP 2173.05(e). The specification does not resolve the ambiguity, because it identifies a fourth quantity that the claims never recite. The specification describes the restoration as an inverse dynamic shift operation, a left shift, that restores the bit width of the results of the element-wise multiplication of the short-bit input feature data and the short-bit weights, using the shift bits of the short-bit input feature data and the shift bits of the short-bit weights, before those results are summed. See paragraphs [0059], [0107], [0108] and [0109]. The bit width of that multiplication result is recited nowhere in claims 1, 11 or 20. In addition, because a bit width can be characterized as restored only by reference to a particular earlier reduction, the recitation of output feature data having "the restored bit width" does not inform one of ordinary skill in the art what bit width the output feature data has, or to what value that bit width is restored. A claim is indefinite when, read in light of the specification, it fails to inform those skilled in the art about the scope of the invention with reasonable certainty. See Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901 (2014); In re Packard, 751 F.3d 1307, 1314 (Fed. Cir. 2014). For purposes of examination, and consistent with paragraphs [0050], [0059] and [0109], the broadest reasonable interpretation of the limitation is taken to be a left shift, performed as part of the convolution operation, that increases the bit width of the element-wise multiplication result by an amount determined by the shift bits of the input feature data and the shift bits of the weights, so that the output feature data has a bit width greater than the reduced bit width of the input feature data.
Claims 2-10 depend, directly or indirectly, from claim 1, and claims 12-19 depend, directly or indirectly, from claim 11. Each of those claims incorporates by reference all of the limitations of the claim from which it depends, including the limitations addressed immediately above, and none of them cures the defect. Claims 2-10 and 12-19 are therefore rejected for the same reason.
Claim 10 recites the limitation "the multiplier specifications included in the display device". There is insufficient antecedent basis for this limitation in the claim. See MPEP 2173.05(e). Neither claim 10, nor claim 9 from which it depends, nor claim 1, recites a multiplier, a specification of a multiplier, or any plurality of multiplier specifications. The claim therefore refers back by definite article to an element that has never been introduced, and it cannot be determined what parameters the size of the data enable region is required to be determined from. The specification does not cure the ambiguity. The identical phrase appears twice in the disclosure, at paragraphs [0126] and [0130], each time preceded by the definite article and each time without definition; the specification never states what a multiplier specification is. The only related teaching is an example in which adjusting the horizontal size of the data enable region by an adjustment coefficient of nine eighths permits 288 pixel data to be processed using sixteen multipliers over eighteen clocks rather than eighteen multipliers over sixteen clocks. See paragraphs [0129] and [0130]. That example is consistent with a specification that is the number of multipliers, but it does not exclude other multiplier parameters such as operand bit width, throughput, or latency, and neither the claim nor the disclosure supplies a basis for choosing among them. See MPEP 2173.03. For purposes of examination, the broadest reasonable interpretation of the limitation is taken to be one or more hardware parameters of the multipliers included in the display device, including at least the number of multipliers available for processing video pixel data, consistent with paragraphs [0129] and [0130].
Claim 17 recites the limitation "the plurality of lightweight convolutional neural networks" in its final clause, and claim 18 recites that same limitation twice. There is insufficient antecedent basis for this limitation in the claims. See MPEP 2173.05(e). Claim 17 depends from claim 16, which depends from claim 11. Claim 11 recites only "a lightweight convolutional neural network", in the singular, and claim 16 introduces only "a plurality of neural network layers", which is a plurality of layers within the single recited network and not a plurality of networks. No plurality of lightweight convolutional neural networks is introduced anywhere in the chain of claims 11, 16 and 17. The corresponding method chain shows that an antecedent was intended and omitted: claim 7, the method counterpart of claim 17, first recites "wherein the method uses a plurality of lightweight convolutional neural networks" and only then refers back to that plurality. Because the apparatus chain contains no such introduction, it cannot be determined whether claim 17 requires a second and further lightweight convolutional neural networks in addition to the single network recited in claim 11, or whether the single network of claim 11 is to be read as satisfying the plurality. Claim 18 depends from claim 17, repeats the same unsupported reference, and does not cure it. For purposes of examination, the broadest reasonable interpretation of the limitation in claims 17 and 18 is taken to be two or more lightweight convolutional neural networks, each generated by reducing the bit width of the weights of a respective original convolutional neural network, consistent with paragraphs [0112] and [0113].
Claim 12 recites "wherein the lightweight convolutional neural network is configured to obtain the lightweight convolutional neural network stored from memory". This limitation renders claim 12 indefinite. The limitation recites the lightweight convolutional neural network as both the thing that performs the obtaining and the thing that is obtained, so that the recited network obtains itself. Earlier in the same claim, the at least one processor is recited as executing the one or more instructions to "store the lightweight convolutional neural network in the memory," so the claim assigns the storing to the processor and the corresponding obtaining to the neural network. It cannot be determined from the claim what structure performs the obtaining, or whether the wherein clause imposes any further limitation on the display device of claim 11 at all. The limitation is also inconsistent with the disclosure, which assigns the obtaining to the display device and not to the neural network: the specification states that the display device stores the lightweight neural network including the short-bit weights in the memory and later calls that stored network when performing neural network computation. See paragraphs [0104], [0105] and [0181]. A claim that is clear on its face may nevertheless be indefinite when a conflict or inconsistency between the claimed subject matter and the specification disclosure renders the scope of the claim uncertain. See MPEP 2173.03; In re Cohn, 438 F.2d 989 (CCPA 1971); In re Moore, 439 F.2d 1232, 1235-36 (CCPA 1971). The recitation "stored from memory" compounds the ambiguity, because the memory from which the network is obtained is recited without an article while the same claim has already recited "the memory," leaving unclear whether the two are the same memory. For purposes of examination, the broadest reasonable interpretation of the limitation is taken to require that the at least one processor execute the one or more instructions to obtain, from the memory recited in claim 11, the lightweight convolutional neural network previously stored in that memory, consistent with paragraphs [0104] and [0181].
Claim 14 recites "determine a shift distance based on values of the first bit width, the second bit width, the third bit width, and the input feature data", and claim 15 recites "perform an element-wise multiplication operation" and "perform a shift operation of restoring a bit width". Each of these limitations is introduced with an indefinite article, although claim 11, from which claims 14 and 15 depend through claim 13, has already recited "determining a shift distance," "performing a convolution operation," and "a shift operation of restoring the bit width." It cannot be determined whether claims 14 and 15 further define the operations already recited in claim 11 or instead require additional and distinct operations, and the claims are therefore indefinite. See MPEP 2173.05(e); MPEP 2173.05(o); In re Kelly, 305 F.2d 909, 916 (CCPA 1962) (the governing consideration is what is a reasonable construction of the language of the claims). The corresponding method claims are drafted so as to avoid the ambiguity: claim 4 recites "wherein the determining of the shift distance comprises" before reciting its sub-steps, and claim 5 recites "wherein the performing of convolution operation comprises" before reciting its sub-steps, so that in each case the sub-steps are expressly tied to an operation already recited in claim 1. Claims 14 and 15 omit those introductory clauses. For purposes of examination, the broadest reasonable interpretation of the limitations of claims 14 and 15 is taken to be further definitions of the shift distance determination and of the convolution operation already recited in claim 11, rather than as requiring a second shift distance, a second element-wise multiplication operation, or a second restoring shift operation, consistent with paragraphs [0091] through [0096] and [0107] through [0109].
Claim Rejections - 35 U.S.C. 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 11-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (hereinafter Li), US 2020/0050429 A1, in view of Hiramatsu et al. (hereinafter Hiramatsu), US 2022/0319443 A1, and further in view of Wu et al. (hereinafter Wu), US 2020/0097816 A1.
Regarding independent claim 1, Li teaches a method, the method comprising: (Li: [0073], "Many machine learning models or neural networks, for example, deep neural network and convolutional neural network, have a multi-layer architecture with tensor processing between the layers."; the processing between the layers of the neural network 100 is carried out as an ordered sequence of acts performed on the data of each layer, which is a method) inputting input data to the lightweight convolutional neural network and performing neural network computations to obtain output data, (Li: [0092], "the first fixed point number 502 is associated with a pixel from one of the p channels of input feature maps at layer t"; the p channels of input feature maps (input data) are presented to layer t of the neural network 100 (inputting to the lightweight convolutional neural network), [0089], "the multiplier 455 performs multiplication of a first number 453 and a second number 457 and output the multiplication result 460 to the adder 470"; the multiply-accumulate circuits 450 carry out the neural network computations and yield the multiplication result 460 (output data)), wherein the neural network computations comprise: determining a shift distance based on a value of input feature data (Li: [0109], "the leading-1 detector 830 determines a position 835 of the left most non-zero bit of Input 830"; the leading-1 position is read out of the value of the first fixed point number 502 (input feature data) itself, [0154], "the exponent generator 2900 receives a leading-1 position 2935 and Input 2924"; the exponent generator is presented with the data value and with the position of the leading-1 bit of that value, [0155], "The look-up tabler matcher 2910 further determines Exp (Input) by searching or looking up the look-up tables from the look-up table database 2960 based on Input 2924 and/or the leading-1 position 2935 in addition to M, E and Stride"; Exp (Input) (a shift distance) is determined from the value of the input feature data and from the position of its leading-1 bit, and is determined before that data is shifted); performing a shift operation to reduce a bit width of the input feature data based on the shift distance (Li: [0157], "The look-up table matcher 2910 further transmits Exp (Input) 2912 to the right shifter 2920"; the shift distance already determined is delivered to the shifter before any shifting of the data occurs, [0157], "The right shifter 2920 outputs an shifted number 2922 to the lowest M-bit selector 2930 by right shifting Input 2924 by a number of bits equal to Exp (Input) 2912 upon receipt of Exp (Input) 2912"; the input feature data is shifted by a number of bit positions equal to Exp (Input) (the shift distance), so the shift operation is performed based on the shift distance rather than the shift distance being counted off a shift already taken, [0157], "The lowest M-bit selector 2930 selects and outputs the lowest (i.e., the right most) M bits of the shifted number 2922 as Man (Input) 2942"; only M bits of the shifted value are kept, so the shift operation leaves the input feature data represented in M bits rather than I bits); performing a convolution operation on the input feature data with the reduced bit width and the weights with the reduced bit width (Li: [0100], "The short bit-width multiplier 550 performs multiplication of the first combination 535 of the first sign and the first mantissa and the second combination 540 of the second sign and the second mantissa"; the multiplication is carried out on Man (A) (the input feature data with the reduced bit width) and Man (B) (the weights with the reduced bit width), [0089], "The adder 470 sums the multiplication result 460 and a third number 490 from a register 480"; the accumulation of those products across the contraction index is the multiply-accumulate operation by which the filters of layer t are applied to the input feature maps), wherein the convolution operation comprises a shift operation of restoring [[the bit width]]a bit width of a result of the convolution operation (interpreted per the 35 U.S.C. 112(b) rejection set forth above) (Li: [0092], "In an embodiment, the multiplier 505 is the multiplier 455 in FIG. 4"; the multiplier 505 that carries the restoration circuit 580 is expressly identified as the multiplier 455 of the multiply-accumulate circuits 450 by which the filters of layer t are applied to the input feature maps, so what the restoration circuit does is done inside the convolution operation rather than after it, [0101], "The adder 560 performs summation of the first exponent 530 (i.e., Exp (A)) and the second exponent 545 (i.e., Exp (B)), and outputs a summation result 575, i.e., Exp (A) + Exp (B), to the restoration circuit 580"; the two exponents are the displacements that were taken out of the operands, [0104], "the restoration circuit 580 is a left shifter. Specifically, the restoration circuit 580 calculates C by left shifting the multiplication result 570 by a number of bits equal to the summation results 575"; the left shift is applied to the multiplication result 570, so the shift that returns the product to full width is part of the multiply-accumulate operation itself rather than a separate later step); and obtaining output feature data with the restored bit width (Li: [0102], "The restoration circuit 580 calculate multiplication of the first fixed point number and the second fixed point number"; the value emitted after the left shift is the full-width product C, [0089], "the third number 490 is a summation result from a previous operation that is stored in the register 480"; the restored products are accumulated in the register 480 and become the data the next layer receives).
Li does not expressly teach a method performed by a display device.
However, Hiramatsu teaches a method performed by a display device (Hiramatsu: [0047], "The television reception device 100 includes a main control unit 201, a bus 202, a storage unit 203, a communication interface (IF) unit 204, an extended interface (IF) unit 205, a tuner/demodulation unit 206"; the television reception device 100 (a display device) is the apparatus that carries out the described processing, [0118], "when the television reception device 100 receives a video stream such as television broadcasting and displays the video streams as a screen on the display unit 219, the neural network 1000 can output an optimal solution of a local dimming pattern which can achieve picture quality improvement such as a high dynamic range for each frame in real time"; the neural network computation is performed by the television reception device 100 itself, on each frame of the incoming video, as part of displaying it).
Because Li and Hiramatsu are analogous art and within the same field of endeavor, specifically the execution of neural network computations on image data by dedicated inference hardware, they address the same problem solving area of obtaining a neural network result for every frame of video within the arithmetic budget of a consumer product, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention, to combine the short bit-width conversion and restoration arithmetic of Li with the frame-rate neural network processing that Hiramatsu performs in the television reception device 100, with a reasonable expectation of success, because Li places its short bit-width arithmetic at the position a neural network's arithmetic already occupies (Li: [0086], "Each APE 440 uses multiply-accumulate circuits (MACs) 450"; the multiply-accumulate circuits 450 are the arithmetic units of the contraction engine that carries out the tensor processing between the layers, [0089], "each MAC 450 includes a multiplier 455 and an adder 470"; the multiplier 455 is the element inside each such circuit, [0092], "In an embodiment, the multiplier 505 is the multiplier 455 in FIG. 4"; the short bit-width multiplier 505 together with its converters and restoration circuit is expressly that same element, so the modification substitutes one multiplier for another and leaves the rest of the network arrangement as it was), which is the simple substitution of one known element for another to obtain predictable results, to teach a method performed by a display device. This modification would have been motivated by the desire to realize a high dynamic range while maintaining constant output power (Hiramatsu: [0003]).
Li and Hiramatsu do not expressly teach obtaining a lightweight convolutional neural network with a reduced bit width of weights.
However, Wu teaches obtaining a lightweight convolutional neural network with a reduced bit width of weights (Wu: [0018], "to generate dynamic fixed-point format weights, a dynamic fixed-point format bias, and dynamic fixed-point format activations for the each CNN layer of the floating pre-trained CNN model 106"; the dynamic fixed-point format weights are generated for every layer of the floating pre-trained convolution neural network model 106 and the network carrying them (a lightweight convolutional neural network) is the network the framework thereafter operates on, [0028], "p represents the quantization bit-width"; the quantization bit-width p is the number of bits the generated weights occupy and is selected below the width of the floating values they replace, so the generated network carries weights of a reduced bit width, [0014], "a memory 108 is used to save a floating pre-trained convolution neural network (CNN) model, a CNN model, and a dynamic fixed-point CNN model"; the dynamic fixed-point CNN model is held in the memory 108 as a network in its own right, so it is a network that can be obtained rather than an operand produced in passing during a multiplication).
Because Li, Hiramatsu, and Wu are analogous art and within the same field of endeavor, specifically the execution of convolutional neural network inference on operands expressed in fewer bits than the values they represent, they address the same problem solving area of running a trained network within the arithmetic and memory budget of a consumer device, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention, to obtain the network on which Li and Hiramatsu’s method performs its short bit-width arithmetic as the dynamic fixed-point network Wu generates in advance from a pre-trained model, with a reasonable expectation of success, because both references reduce the same object, the trained weight of a convolutional layer, to a representation of fewer bits, and Wu produces that representation once per layer ahead of inference rather than once per multiplication (Wu: [0018], "to generate dynamic fixed-point format weights, a dynamic fixed-point format bias, and dynamic fixed-point format activations for the each CNN layer of the floating pre-trained CNN model 106"; the generation is performed per layer over the whole pre-trained model, so the network reaches the arithmetic already reduced), to teach obtaining a lightweight convolutional neural network with a reduced bit width of weights. This modification would have been motivated by the desire to spend the conversion effort once per layer instead of once per multiplication and to hold the network in less memory (Wu: [0014]).
Regarding dependent claim 2, Li, in view of Hiramatsu and Wu, teach the method of claim 1, further comprising: obtaining an original convolutional neural network (Wu: [0014], "a memory 108 is used to save a floating pre-trained convolution neural network (CNN) model, a CNN model, and a dynamic fixed-point CNN model"; the floating pre-trained convolution neural network model (an original convolutional neural network) is held alongside the dynamic fixed-point CNN model generated from it, so the original network is obtained and the generated network is stored); and generating the lightweight convolutional neural network by reducing a bit width of weights of the original convolutional neural network and storing the lightweight convolutional neural network in a memory of the display device, wherein the obtaining of the lightweight convolutional neural network comprises obtaining the lightweight convolutional neural network from the memory (Wu: [0018], "to generate dynamic fixed-point format weights, a dynamic fixed-point format bias, and dynamic fixed-point format activations for the each CNN layer of the floating pre-trained CNN model 106"; the generation acts on the weights of each layer of the pre-trained model, which is reducing a bit width of weights of the original convolutional neural network, and the stored dynamic fixed-point CNN model is what is read back from the memory 108 when the framework runs, Hiramatsu: [0049], "The storage unit 203 stores operation programs and operation setting values of the television reception device 100"; storage unit 203 (a memory) of the television reception device 100 (of the display device) in which the generated lightweight convolutional neural network is stored and obtained).
Regarding dependent claim 3, Li, in view of Hiramatsu and Wu, teach the method of claim 1, wherein the weights with the reduced bit width and the input feature data with the reduced bit width comprise information bits representing bits with a bit width reduced from original data (Li: [0096], "The mantissa is denoted by Man (Input), and the bit width of the mantissa is denoted by M"; Man (Input) (information bits) holds the M bits selected out of the I bits of the unconverted value, [0100], "The short bit-width multiplier 550 is named so because both operands have shorter bit widths than their original fixed point numbers"; the same conversion is applied to the filter operand and to the input feature map operand alike), and shift bits representing bits including shift distance information (Li: [0096], "The exponent is denoted by Exp (Input), and the bit width of the exponent is denoted by E"; Exp (Input) (shift bits) is carried alongside the mantissa in E bits, [0157], "The right shifter 2920 outputs an shifted number 2922 to the lowest M-bit selector 2930 by right shifting Input 2924 by a number of bits equal to Exp (Input) 2912 upon receipt of Exp (Input) 2912"; what those E bits carry is the number of bit positions by which the value was shifted, which is the displacement information needed to undo the reduction).
Regarding dependent claim 4, Li, in view of Hiramatsu and Wu, teach the method of claim 3, wherein the determining of the shift distance comprises: identifying a first bit width, which is the bit width of the input feature data (Li: [0095], "before conversion, the fixed point number has a bit width denoted by I"; I (a first bit width) is the width of the value as it stands before the conversion runs, [0112], "the leading-1 detector 830 obtains the bit width of Input 805, i.e., I, from the parameter database"; I is read out expressly as an input to the conversion); determining a second bit width, which is the bit width of the shift bits (Li: [0096], "The exponent is denoted by Exp (Input), and the bit width of the exponent is denoted by E"; E (a second bit width) is the width allotted to Exp (Input) (the shift bits)); determining a third bit width, which is the bit width of the information bits (Li: [0096], "The mantissa is denoted by Man (Input), and the bit width of the mantissa is denoted by M"; M (a third bit width) is the width allotted to Man (Input) (the information bits)); and determining a shift distance based on values of the first bit width, the second bit width, the third bit width, and the input feature data (Li: [0112], "the exponent generator 840 obtains the bit width of Input 805, i.e., I, the bit width of the exponent of Input 805 or Input 822, i.e., E, the bit width of the mantissa of Input 805 or Input 822, i.e., M from the parameter database 850, and the value of the stride number, i.e., Stride"; I (the first bit width), E (the second bit width) and M (the third bit width) are read into the exponent generator 840 as inputs to its determination of Exp (Input), [0155], "The look-up tabler matcher 2910 further determines Exp (Input) by searching or looking up the look-up tables from the look-up table database 2960 based on Input 2924 and/or the leading-1 position 2935 in addition to M, E and Stride"; the exponent generator determines Exp (Input) (a shift distance) from the value of the input feature data and the position of its leading-1 bit taken together with M and E, so the first bit width, the second bit width, the third bit width and the input feature data are each inputs to the determination, [0109], "the leading-1 detector 830 determines a position 835 of the left most non-zero bit of Input 830"; what the input feature data contributes to the determination is the position of its own leading-1 bit).
Regarding dependent claim 5, Li, in view of Hiramatsu and Wu, teach the method of claim 3, wherein the performing of convolution operation comprises: performing an element-wise multiplication operation by using information bits of the input feature data with the reduced bit width and information bits of the weights with the reduced bit width (Li: [0100], "The short bit-width multiplier 550 performs multiplication of the first combination 535 of the first sign and the first mantissa and the second combination 540 of the second sign and the second mantissa"; the multiplier is presented with Man (A) (the information bits of the input feature data with the reduced bit width) and Man (B) (the information bits of the weights with the reduced bit width) and with nothing else); and performing a shift operation of restoring a bit width by using shift bits of the input feature data with the reduced bit width and shift bits of the weights with the reduced bit width (Li: [0101], "The adder 560 performs summation of the first exponent 530 (i.e., Exp (A)) and the second exponent 545 (i.e., Exp (B)), and outputs a summation result 575, i.e., Exp (A) + Exp (B), to the restoration circuit 580"; Exp (A) and Exp (B) are exactly the shift bits taken from the two operands, [0104], "the restoration circuit 580 calculates C by left shifting the multiplication result 570 by a number of bits equal to the summation results 575"; the restoring left shift is driven by nothing other than the sum of those two shift-bit fields).
Regarding dependent claim 6, Li, in view of Hiramatsu and Wu, teach the method of claim 1, wherein the lightweight convolutional neural network comprises a plurality of neural network layers (Li: [0073], "Many machine learning models or neural networks, for example, deep neural network and convolutional neural network, have a multi-layer architecture with tensor processing between the layers."; the neural network 100 (the lightweight convolutional neural network) is built of layer (t-1), layer t and layer (t+1), which are a plurality of neural network layers), and the neural network computations including bit width reduction and restoration respectively correspond to the plurality of neural network layers (Li: [0115], "the parameters 900 including without limitation, the bit width of the fixed point number denoted by I, the bit width of the mantissa denoted by M, the bit width of the exponent denoted by E (t, c, a, b), the bit width of the sign denoted by S, a value of the base number denoted by W, and a value of the stride number denoted by Stride are subject to change with respect to layer t, channel c, and effective coordinate (a, b)"; the widths that govern how far the value is reduced and how far it is shifted back are indexed by the layer, so a separate reduction and restoration is defined for each layer, [0116], "the precision requirement for the multiplication of the two fixed point numbers increases with the increase of the layer index, t, from the middle layer to the last layer of the neural network"; Li ties the amount of reduction to the layer being processed rather than applying one setting to the whole network).
Regarding independent claim 11, it is a display device claim that is substantially the same as the method of claim 1. Thus, claim 11 is rejected for the same reasons as claim 1. In addition, Hiramatsu teaches a display device comprising: a communication interface; memory storing one or more instructions; and at least one processor configured to execute the one or more instructions stored in the memory to (Hiramatsu: [0047], "The television reception device 100 includes a main control unit 201, a bus 202, a storage unit 203, a communication interface (IF) unit 204, an extended interface (IF) unit 205, a tuner/demodulation unit 206"; the television reception device 100 (a display device) carries the communication interface (IF) unit 204 (a communication interface) as one of its enumerated parts, [0048], "The controller is configured as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), or the like"; the controller of the main control unit 201 (at least one processor) is what executes those stored programs, [0049], "The storage unit 203 stores operation programs and operation setting values of the television device 100"; so the operation instructions the controller executes are the instructions stored in the storage unit 203).
Regarding dependent claims 12-13 and 16, these are display device claims that are substantially the same as the method claims 2-3 and 6, respectively. Thus, claims 12-13 and 16 are rejected for the same reasons as claims 2-3 and 6.
Regarding dependent claim 14, Li, in view of Hiramatsu and Wu, teach the display device of claim 13, wherein the at least one processor is further configured to execute the one or more instructions to: identify a first bit width, which is the bit width of the input feature data (Li: [0095], "before conversion, the fixed point number has a bit width denoted by I"; I (a first bit width) is the width of the value before conversion); determine a second bit width, which is the bit width of the shift bits (Li: [0096], "The exponent is denoted by Exp (Input), and the bit width of the exponent is denoted by E"; E (a second bit width) is the width allotted to the exponent field); determine a third bit width, which is the bit width of the information bits (Li: [0096], "The mantissa is denoted by Man (Input), and the bit width of the mantissa is denoted by M"; M (a third bit width) is the width allotted to the mantissa field); and determine [[a]]the shift distance (interpreted per the 35 U.S.C. 112(b) rejection set forth above) based on values of the first bit width, the second bit width, the third bit width, and the input feature data (Li: [0112], "the exponent generator 840 obtains the bit width of Input 805, i.e., I, the bit width of the exponent of Input 805 or Input 822, i.e., E, the bit width of the mantissa of Input 805 or Input 822, i.e., M from the parameter database 850, and the value of the stride number, i.e., Stride"; I (the first bit width), E (the second bit width) and M (the third bit width) are read into the exponent generator 840 as inputs to its determination of Exp (Input) rather than being derived from that determination, [0155], "The look-up tabler matcher 2910 further determines Exp (Input) by searching or looking up the look-up tables from the look-up table database 2960 based on Input 2924 and/or the leading-1 position 2935 in addition to M, E and Stride"; the exponent generator determines Exp (Input) (the shift distance) from the value of the input feature data and the position of its leading-1 bit taken together with M and E, so the values of the first bit width, the second bit width, the third bit width and the input feature data are each inputs to the determination, [0157], "The right shifter 2920 outputs an shifted number 2922 to the lowest M-bit selector 2930 by right shifting Input 2924 by a number of bits equal to Exp (Input) 2912 upon receipt of Exp (Input) 2912"; the shift distance so determined is the same one by which the input feature data is then shifted rather than a second, separately computed one).
Regarding dependent claim 15, Li, in view of Hiramatsu and Wu, teach the display device of claim 13, wherein the at least one processor is further configured to execute the one or more instructions to perform the convolution operation by: [[perform]]performing (interpreted per the 35 U.S.C. 112(b) rejection set forth above) an element-wise multiplication operation by using information bits of the input feature data with the reduced bit width and information bits of the weights with the reduced bit width (Li: [0100], "The short bit-width multiplier 550 performs multiplication of the first combination 535 of the first sign and the first mantissa and the second combination 540 of the second sign and the second mantissa"; the multiplier receives Man (A) (the information bits of the input feature data with the reduced bit width) and Man (B) (the information bits of the weights with the reduced bit width), and the multiplication it performs is the element-wise product that the accumulation of the multiply-accumulate operation is taken over); and [[perform]]performing (interpreted per the 35 U.S.C. 112(b) rejection set forth above) a shift operation of restoring a bit width by using shift bits of the input feature data with the reduced bit width and shift bits of the weights with the reduced bit width (Li: [0101], "The adder 560 performs summation of the first exponent 530 (i.e., Exp (A)) and the second exponent 545 (i.e., Exp (B)), and outputs a summation result 575, i.e., Exp (A) + Exp (B), to the restoration circuit 580"; Exp (A) and Exp (B) are the shift bits of the two converted operands, [0104], "the restoration circuit 580 calculates C by left shifting the multiplication result 570 by a number of bits equal to the summation results 575"; the left shift restores the product to the width it would have had if neither operand had been shortened).
Regarding independent claim 20, it is a non-transitory computer-readable recording medium claim that is substantially the same as the method of claim 1. Thus, claim 20 is rejected for the same reason as claim 1. In addition, Li teaches a non-transitory computer-readable recording medium having recorded thereon a program for causing a display device to perform a method, the method including: (Li: [0225], "Non-volatile media include, for example, solid state, optical or magnetic disks, such as the storage device 4910. Volatile media include dynamic memory, such as the main memory 4906. Non-volatile and volatile media are considered non-transitory"; the storage device 4910 (a non-transitory computer-readable recording medium) is identified as non-transitory in terms, [0224], "The computer system 4900 may be suitable to implement methods as described herein in response to the processor 4904 executing one or more sequences of one or more instructions contained in, e.g., the main memory 4906. Such instructions may be read into main memory 4906 from another computer-readable medium, such as the storage device 4910"; the instructions that make the described method run are recorded on that medium and read from it, which is a program recorded thereon for causing the machine to perform the method).
Claims 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Hiramatsu and Wu, as applied in the rejections of claims 6 and 16 above, and further in view of Wang et al. (hereinafter Wang) “Deep Network Interpolation for Continuous Imagery Effect Transition” 2019.
Regarding dependent claim 7, Li, in view of Hiramatsu and Wu, teach the method of claim 6, wherein the performing of the neural network computations comprises performing the neural network computations including the bit width reduction and restoration (Wu: [0018], "to generate dynamic fixed-point format weights, a dynamic fixed-point format bias, and dynamic fixed-point format activations for the each CNN layer of the floating pre-trained CNN model 106"; the floating pre-trained CNN model 106 (an original convolutional neural network) is the network the process is given, and what the process yields is that network's weights carried in the shorter dynamic fixed-point format, [0014], "a memory 108 is used to save a floating pre-trained convolution neural network (CNN) model, a CNN model, and a dynamic fixed-point CNN model"; the reduced network is held as a model in its own right alongside the original it was produced from).
Li, Hiramatsu, and Wu do not expressly teach wherein the method uses a plurality of lightweight convolutional neural networks, and the performing of the neural network computations by using the plurality of lightweight convolutional neural networks.
However, Wang teaches wherein the method uses a plurality of lightweight convolutional neural networks, and the performing of the neural network computations comprises performing the neural network computations by using the plurality of lightweight convolutional neural networks (Wang: page 1694, Section 3.1, "Consider two networks GA and GB with the same structure, achieving different effects A and B, respectively"; two convolutional networks built to one structure, each holding its own trained parameters, are kept at the same time and are the networks the method proceeds from, which is a plurality of such networks (wherein the method uses a plurality of lightweight convolutional neural networks), page 1694, Section 3.1, "the proposed Deep Network Interpolation (DNI)…Generally, DNI can be extended for N models"; the network interpolation operations DNI (and the performing of neural network computations comprises performing the neural network computations) treatment is stated for any number N of those networks, so the plurality is not confined to two (by using the plurality of lightweight convolutional neural networks)).
Because Li, Hiramatsu, Wu, and Wang are analogous art and within the same field of endeavor, specifically the construction and operation of convolutional neural networks that a device applies to an image it is to output, they address the same problem solving area of obtaining a wanted picture result from convolutional networks whose arithmetic the device can afford to carry, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention, to keep two or more of the reduced-bit-width convolutional neural networks of the combination rather than one and to perform the neural network computations of claim 6 by using them, with a reasonable expectation of success, because the several networks share one structure and differ only in the values their weights take, so each is run by the same short bit width arithmetic already set forth for claim 1, to teach wherein the method uses a plurality of lightweight convolutional neural networks, and the performing of the neural network computations comprises performing the neural network computations including the bit width reduction and restoration by using the plurality of lightweight convolutional neural networks. This modification would have been motivated by the desire to obtain more than one picture result from the device, each network being trained to a different effect (Wang: page 1692, Section 1).
Regarding dependent claim 8, Li, in view of Hiramatsu, Wu, and Wang, teach the method of claim 7, further comprising combining weights with a reduced bit width of the plurality of lightweight convolutional neural networks based on a predefined criterion, wherein the performing of the neural network computations comprises performing a convolution operation by using a combination of the weights with the reduced bit width of the plurality of lightweight convolutional neural networks (Wu: [0026] operation include combining weights with integer and fractional word length, Wang: page 1694, Section 3.1, "DNI interpolates all the corresponding parameters of these two models to derive a new interpolated model"; the parameters standing in corresponding positions across the several networks are taken together into one derived set, which is combining them, page 1694, Section 3.1, "The interpolation is performed on all the layers with parameters in the networks, including convolutional layers and normalization layers"; the operation reaches every parameterized layer, page 1694, Section 3.1, "Convolutional layers have two parameters, namely weights (filters) and biases"; the parameters so combined at a convolutional layer are its weights, page 1694, Section 3.1, "In other words, it is a convex combination of the parameter vectors"; the coefficients that govern the combination, set out in equation (2) immediately above that sentence, are admitted only if they are non-negative and sum to one, which is what a convex combination requires, a rule settled before any particular combination is formed and therefore a criterion defined in advance of the combining, page 1693, Section 1, "Performing feed-forward operations on these interpolated models using the same input"; the forward pass that follows is carried out with those combined filters, so the convolution is performed by using the combination).
Because Li, Hiramatsu, Wu, and Wang are analogous art and within the same field of endeavor, specifically the construction and operation of convolutional neural networks that a device applies to an image it is to output, they address the same problem solving area of obtaining a wanted picture result from convolutional networks whose arithmetic the device can afford to carry, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention, to form the weights standing in corresponding positions across the plurality of claim 7 into a single set of weights under the convex-combination rule Wang fixes for the coefficients, and to perform the convolution operation of the combination upon that single set, with a reasonable expectation of success, because the combining settles only what values the weights take and prescribes nothing about the format in which they are held or the multiplication and restoring shift afterwards performed upon them (Wang: page 1694, Section 3.1, "The interpolation is performed on all the layers with parameters in the networks, including convolutional layers and normalization layers"; the operation reaches the weights of every parameterized layer and prescribes nothing about the arithmetic those weights afterwards enter), to teach further comprising combining weights with a reduced bit width of the plurality of lightweight convolutional neural networks based on a predefined criterion, wherein the performing of the neural network computations comprises performing a convolution operation by using a combination of the weights with the reduced bit width of the plurality of lightweight convolutional neural networks. This modification would have been motivated by the desire to draw on all of the networks at once without adding to the per-frame arithmetic the device performs, the several networks being reduced to one set of weights before the convolution operation is reached (Wang: page 1694, Section 3.1).
Regarding dependent claims 17-18, these are display device claims that are substantially the same as the method of claims 7-8. Thus, claims 17-18 are rejected for the same reasons as claims 7-8.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view Hiramatsu and Wu, as applied in the rejections of claims 1 and 11, respectively above, and further in view of Takasu et al. (hereinafter Takasu), US 6,366,263 B1.
Regarding dependent claim 9, Li, in view of Hiramatsu and Wu, teach all the elements of claim 1.
Li, Hiramatsu, and Wu do not expressly teach identifying a horizontal raster size and a vertical raster size of a video frame; and adjusting a size of a data enable region, which is a region with valid pixel data, based on the horizontal raster size and the vertical raster size.
However, Takasu teaches identifying a horizontal raster size and a vertical raster size of a video frame (Takasu: 2:4-5, "the size of the raster in the horizontal direction will be referred to as a horizontal raster size, and the size of the raster in the vertical direction as a vertical raster size"; Takasu names the two dimensions of the raster of the displayed frame in the terms the claim uses, 3:6-8, "the information of the Hraster (horizontal raster size) and Vraster (vertical raster size) of the raster which is being displayed currently is obtained"; both raster dimensions of the frame currently being displayed are read out before the size is varied, which is identifying them); and adjusting a size of a data enable region, which is a region with valid pixel data, based on the horizontal raster size and the vertical raster size (Takasu: 2:6-10, "the active ratio refers to the occupied ratio (horizontal active ratio) of the display image in the horizontal direction with respect to the horizontal raster size, and the occupied ratio (vertical active ratio) of the display image in the vertical direction with respect to the vertical raster size"; the display image is the portion of the raster that carries picture content (a region with valid pixel data) and its horizontal and vertical extents are defined as ratios taken with respect to the horizontal raster size and the vertical raster size respectively, 2:25-27, "the horizontal size of the display image G formed within the raster RS is also enlarged"; the extent of that valid-pixel region is changed as the raster dimension it is referred to changes, which is adjusting its size based on the raster sizes).
Because Li, Hiramatsu, Wu, and Takasu are analogous art and within the same field of endeavor, specifically the formation and processing of the picture carried in the raster of a video frame by a device that displays it, they address the same problem solving area of fitting the per-frame work a display device must perform into the fixed time and hardware a frame allows, accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention, to read out the horizontal and vertical raster sizes of the frame and size the valid-pixel region of that frame against them as Takasu directs, in the display device of the combination that performs the neural network computation on each displayed frame, with a reasonable expectation of success, because Takasu's sizing acts on the raster of the frame before the picture is delivered to the screen and leaves the arithmetic performed on the picture untouched, so it may be applied to the combined device without disturbing the short bit-width computation Li performs, to teach identifying a horizontal raster size and a vertical raster size of a video frame and adjusting a size of a data enable region based on them. This modification would have been motivated by the desire to ease difficulty to control how much valid pixel data the device must carry through each frame, which is the quantity that sets the per frame arithmetic load of the neural network computation the combination performs on every displayed frame (Takasu: 3:3-6).
Regarding dependent claim 19, it is a display device claim that is substantially the same as the method of claim 9. Thus, claim 19 is rejected for the same reason as claim 9.
Allowable Subject Matter
Claim 10 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The closest prior arts found when taken individually or in combination do not expressly teach or render obvious the limitations recited in dependent claim 10 when taken in the context of the claims as a whole.
At best the closest prior arts uncovered, specifically, Li (US 2020/0050429 A1) disclose: A multiplier for calculating a multiplication of a first fixed point number and a second fixed point number comprises a converter and a restoration circuit. The converter is configured to convert the first fixed point number to a sign, a mantissa, and an exponent. At least one of a bit width of the sign, a bit width of the mantissa, and a bit width of the exponent is dynamically configured based on a position of a layer associated with the first fixed point number in a neural network, a position of a pixel in an input feature map associated with the first fixed point number, and/or a channel associated with the first fixed point number. The restoration circuit is configured to calculate the multiplication based on the sign, the mantissa, the exponent, and the second fixed point number (Abstract); Hiramatsu (US 2022/0319443 A1) disclose: An image processing device that realizes local dimming control and push-up control of a display device using an artificial intelligence function is provided. The image processing device includes a trained neural network model that estimates a local dimming pattern representing light emitting states of light source units corresponding to a plurality of areas divided from a display area of an image display unit for a target display image, and a control unit that controls the light emitting states of the light source units on the basis of the local dimming pattern estimated by the trained neural network model for the target display image displayed on the image display unit (Abstract); Wu (US 2020/0097816 A1) disclose: A system for operating a floating-to-fixed arithmetic framework includes a floating-to-fix arithmetic framework on an arithmetic operating hardware such as a central processing unit (CPU) for computing a floating pre-trained convolution neural network (CNN) model to a dynamic fixed-point CNN model. The dynamic fixed-point CNN model is capable of implementing a high performance convolution neural network (CNN) on a resource limited embedded system such as mobile phone or video cameras (Abstract); and Takasu (US 6,366,263 B1) disclose: The size of an image displayed on a tube surface is varied while maintaining the aspect ratio thereof at a constant. First, if there is information of an arbitrary horizontal-size variable parameter and a vertical-size variable parameter because of parameters supplied previously by adjustments, the horizontal/vertical size of a raster formed at that time can be obtained, and the aspect ratio thereof can be obtained on the basis of the horizontal/vertical size of the raster. Then, during a zoom mode, the vertical-size variable parameter corresponding to the updated horizontal-size variable parameter is computed so that the detected aspect ratio becomes constant, and the raster size is changed in accordance with these values (Abstract).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
SON et al., US 2022/0202200 A1 (Jun. 25, 2020) (ABSTRACT A processor-implemented neural network processing method includes: obtaining a kernel bit-serial block corresponding to first data of a weight kernel of a layer in a neural network; generating a feature map bit-serial block based on second data of one or more input feature maps of the layer; and generating at least a portion of an output feature map by performing a convolution operation of the layer using a bitwise operation between the kernel bit-serial block and the feature map bit-serial block).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights