DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
When the broadest reasonable interpretation of a claim covers transitory media or a signal per se, the claim must be rejected under 35 US.C. § 101 as covering non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 1356-57 (Fed. Cir. 2007) (transitory embodiments are not directed to statutory subject matter)
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim is directed to a “ One or more computer-readable storage media storing instructions thereon that when executed cause one or more processors to: …” which may include transitory embodiments.
Claims 1-20 are rejected as being directed toward patent ineligible subject matter under 35 U.S.C. 101, under the “Revised Patent Subject Matter Eligibility Guidance” issued on January 7, 2019 (Federal Register, Vol. 84, No. 4, 50).
The claims are directed to statutory categories of methods, apparata, articles of manufacture (under Step 1).
Upon analysis of the present claims under the broadest reasonable interpretation (under Step 2A, prong one), the claims appear to recite a judicial exception, an abstract idea, directed to logical concepts, mathematical relationships, following rules or instructions, mental processes (observation, evaluation, judgment, and allusions to empirical determinations without limitation to particular empirical equations or algorithms) of manipulating and filtering video data using neural networks.
The claims include several categories of this abstract idea: information (a current block of video data … a filtered current block of video data, … an attention map … feature data … scale factor … ratio value … backbone blocks … a query matrix … a key matrix … a subsequent block …), collecting information (receiving); outputting information (generating … filtering … decoding … encoding …), and/or analyzing information at a high degree of algorithmic generality (generating … filtering … modifying … scaling … decoding … encoding …). These categories have been identified as abstract ideas by the Federal Circuit as summarized in Electric Power Group, LLC v. ALSTOM SA, 830 F. 3d 1350, 1354 (Fed. Cir. 2016):
Information as such is an intangible. See Microsoft Corp. v. AT & T Corp., 550 U.S. 437, 451 n.12, 127 S.Ct. 1746, 167 L.Ed.2d 737 (2007); Bayer AG v. Housey Pharm., Inc., 340 F.3d 1367, 1372 (Fed. Cir. 2003). Accordingly, we have treated collecting information, including when limited to particular content (which does not change its character as information), as within the realm of abstract ideas. See, e.g., Internet Patents, 790 F.3d at 1349; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015); Content Extraction & Transmission LLC v. Wells Fargo Bank, Nat'l Ass'n, 776 F.3d 1343, 1347 (Fed. Cir. 2014); Digitech Image Techs., LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344, 1351 (Fed. Cir. 2014); CyberSource Corp. 1354*1354 v. Retail Decisions, Inc., 654 F.3d 1366, 1370 (Fed. Cir. 2011). In a similar vein, we have treated analyzing information by steps people go through in their minds, or by mathematical algorithms, without more, as essentially mental processes within the abstract-idea category. See, e.g., TLI Commc'ns, 823 F.3d at 613; Digitech, 758 F.3d at 1351; SmartGene, Inc. v. Advanced Biological Labs., SA, 555 Fed.Appx. 950, 955 (Fed. Cir. 2014); Bancorp Servs., L.L.C. v. Sun Life Assurance Co. of Canada (U.S.), 687 F.3d 1266, 1278 (Fed. Cir. 2012); CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372 (Fed. Cir. 2011); SiRF Tech., Inc. v. Int'l Trade Comm'n, 601 F.3d 1319, 1333 (Fed. Cir. 2010); see also Mayo, 132 S.Ct. at 1301; Parker v. Flook, 437 U.S. 584, 589-90, 98 S.Ct. 2522, 57 L.Ed.2d 451 (1978); Gottschalk v. Benson, 409 U.S. 63, 67, 93 S.Ct. 253, 34 L.Ed.2d 273 (1972). And we have recognized that merely presenting the results of abstract processes of collecting and analyzing information, without more (such as identifying a particular tool for presentation), is abstract as an ancillary part of such collection and analysis. See, e.g., Content Extraction, 776 F.3d at 1347; Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 715 (Fed. Cir. 2014).
Upon consideration of the record (under Step 2A, prong two), Examiner did not find that the additional elements of the present claims integrate the judicial exception into a practical application of that judicial exception “in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception.” The additional elements, when considered individually or in a claim as a whole, “ A method of processing video data … A device for processing video data, the device comprising: one or more memories configured to store the video data; and processing circuitry coupled to the one or more memories, wherein the processing circuitry is configured to: … One or more computer-readable storage media storing instructions thereon that when executed cause one or more processors to: …”, do not seem to reflect a substantive improvement in the functioning of a computer, or an improvement to other technology or technical field under the standards of the present judicial guidance; do not seem use a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim (a general purpose processor is not a particular machine); do not seem to effect a transformation or reduction of a particular article to a different state or thing (data is not an article).
This is further evidenced in that the additional elements, merely recite the words ‘‘apply it’’ (or an equivalent such as generate) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea; adds insignificant extra-solution activity to the judicial exception (i.e. obtaining, analyzing, transforming, or outputting information for use with the judicial exception as in CyperSource and Mayo); do no more than generally link the use of a judicial exception to a particular technological environment or field of use (i.e. linked to a computer or other well-established activities in the art such as decoding and encoding of information).
Substantially similar subject matter has been found ineligible in In re Prater, 415 F.2d 1393, 1404-05, 162 USPQ 541, 550-51 (CCPA 1969) (An abstract idea rejection under 35 U.S.C. 101, for claiming a process of analyzing data by selecting the data to be analyzed and by subjecting the data to a mathematical manipulation); Gottschalk v. Benson, 409 U.S. 63, 175 U.S.P.Q. 673 (1972) (using common computing elements in conversion of numerical information is ineligible); RecogniCorp, LLC v. Nintendo Co., Ltd., 855 F. 3d 1322 (Fed. Cir. 2017) (Encoding and decoding of an image is an abstract concept long utilized to transmit information, and addition of a mathematical equation that simply changes the data into other forms of data cannot render it patent eligible); Intellectual Ventures I LLC v. Capital One Fin. Corp., 850 F.3d 1332, 1340-41 (Fed. Cir. 2017) (organizing, displaying, and manipulating data encoded for human and machine readability is directed to an abstract concept).
Finally, the claimed elements, when considered individually and in combination (under step 2B), do not seem to provide an Inventive Concept that is “significantly more” than the ineligible subject matter. The claims simply append well-understood, routine, conventional activities previously known to the industry to the judicial exception, at a high level of generality (A method of processing video data …). Materially, the claimed abstract idea broadly manipulates video data (using a neural network which is not limited to a particular algorithm) in a manner that is indifferent to a practical application. As noted in original Claims 9-10, the claimed limitations are indifferent to the application being encoding or decoding, and indifferent to data being pictures or video.
The claims should be amended to include meaningful limitations within the technical field, such as complete details of transforming recorded video into a bitstream that is stored or transmitted.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20240267543 to Zou (“Zou”) in view of US 20250119541 to Li (“Li”).
Regarding Claim 1: “A method of processing video data, the method comprising:
receiving, with a neural network in-loop filter (NN-ILF), a current block of video data of a current picture; and (“In the case of in-loop filters, a filter applied on one block in the currently-encoded or currently-decoded frame … having the NN replacing all traditional in-loop filters;” Zou, Paragraphs 308-312 and Fig. 8. See similarly in Li, Paragraph 332.)
filtering, with the NN-ILF, the current block of video data to generate a filtered current block of video data, (“In the case of in-loop filters, a filter applied on one block in the currently-encoded or currently-decoded frame … having the NN replacing all traditional in-loop filters;” Zou, Paragraphs 308-312 and Fig. 8. See similarly in Li, Paragraph 332.))
wherein filtering the current block of video data comprises: … generating, with an attention block [of the NN-ILF], an attention map indicative of a correlation between elements of features of the current block of video data; (Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, the attention map is a tensor data structure. Prior art teaches this feature: “providing a first set of feature tensors for first set of scales of the one or more different scales as an input to an attention block” Zou, Paragraph 24. Also, “computing an attention map between the 1D feature tensor and 3D tensor feature maps by using a scaled dot product attention.” Zou, Paragraph 11. See similarly in Li, Paragraph 332.))
modifying, with the attention block [of the NN-ILF], the attention map based on a size of blocks used for training [the NN-ILF] to generate a modified attention map; (Prior art teaches “generating a second set of smaller feature tensors … to generate a first downscaled [modified] feature map” Zou, Paragraph 24. “Consider an example, in which the input data is an image, when the input image is a 128x128x3 RGB image [block] (with horizontal size of 128 pixels, vertical size of 128 pixels, and 3 channels for the Red, Green, Blue color components), and when the encoder downsamples the input tensor [attention map] by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or 'shape') 64x64x32 (e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used.” Zou, Paragraph 353. In this case, modification is a downsampling based on the size of the input image blocks and the block sizes used by the neural network. See similarly in Li Paragraphs 348, 363, and statement of motivation below.)
generating, with the attention block [of NN-ILF], feature data based on the modified attention map; and (“generating a second set of smaller feature tensors … to generate a first downscaled [modified] feature map” … e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used.” Zou, Paragraphs 24, 353.)
filtering, with the NN-ILF, the current block of video data based on the feature data to generate the filtered current block of video data.” (“video codecs may use a set of filters to enhance the visual quality of the predicted and error-compensated visual content and can be applied either in-loop or out-of-loop, or both. … having the NN replacing all traditional in-loop filters;” Zou, Paragraph 312. See similarly in Li, Paragraph 332.)
Note that Zou separately teaches details of how to apply neural networks to processing image blocks of different sizes and that neural network processing can be applied to various steps of image block processing including loop filtering “NN-ILF”. While Zou does not provide an example that combines the details of neural network processing to the loop filter (or other video block coding features), Paragraphs 311-316 indicate that the neural network processing techniques are intended to apply to such video coding features.
Cumulatively, Li indicates that neural network filters were known to apply to video blocks: “The method includes applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample, wherein the NN filter includes an NN filter model generated based on partitioning information of the video unit,” which defines block size. Li, Paragraph 4 and Fig. 30.
Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to supplement the teachings of Zou to apply the steps of neural network processing to a loop filter based on block size, as taught in Li, for “performing a conversion between a video media file and the bitstream based on the filtered sample.” Li, Paragraph 4.
Finally, in reviewing the present application, there does not seem to be objective evidence that the claim limitations are particularly directed to: addressing a particular problem which was recognized but unsolved in the art, producing unexpected results at the level of the ordinary skill in the art, or any other objective indicators of non-obviousness.
Regarding Claim 2: “The method of claim 1, wherein modifying the attention map comprises modifying the attention map utilizing only linear operations.” (For example, “the encoder downsamples the input tensor [attention map] by 2 … dimensions may differ depending on the convention which is used.” Zou, Paragraph 353. See similarly in Li Paragraphs 348, 363, and statement of motivation in Claim 1.)
Regarding Claim 3: “The method of claim 1, wherein modifying the attention map comprises: determining a scale factor based on a ratio of a number of samples in the current block of video data and a number of samples in a block used for training; and scaling the attention map based on the scale factor to generate the modified attention map.” (“Consider an example, in which the input data is an image, when the input image is a 128x128x3 RGB image [block] (with horizontal size of 128 pixels, vertical size of 128 pixels, and 3 channels for the Red, Green, Blue color components), and when the encoder downsamples the input tensor [attention map] by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or 'shape') 64x64x32 (e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used.” Zou, Paragraph 353. In this case, modification is a downsampling based on the size of the input image blocks and the block sizes used by the neural network. See similarly in Li Paragraphs 348, 363, and statement of motivation in Claim 1.)
Regarding Claim 4: “The method of claim 3, wherein determining the scale factor comprises: determining a ratio value based on the ratio of the number of samples in the current block of video data and the number of samples in the block used for training; and multiplying the ratio value with a number greater than one to determine the scale factor.” (“Consider an example, in which the input data is an image, when the input image is a 128x128x3 RGB image [block] (with horizontal size of 128 pixels, vertical size of 128 pixels, and 3 channels for the Red, Green, Blue color components), and when the encoder downsamples the input tensor [attention map] by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or 'shape') 64x64x32 (e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used.” Zou, Paragraph 353. In this case, the scale feactor is 2 which is greater than 1. See similarly in Li Paragraphs 348, 363, and statement of motivation in Claim 1.)
Regarding Claim 5: “The method of claim 1, wherein modifying the attention map comprises: down-sampling the attention map to match a resolution of the blocks used for training.” (“Consider an example, in which the input data is an image, when the input image is a 128x128x3 RGB image [block] (with horizontal size of 128 pixels, vertical size of 128 pixels, and 3 channels for the Red, Green, Blue color components), and when the encoder downsamples the input tensor [attention map] by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or 'shape') 64x64x32 (e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used.” Zou, Paragraph 353. See similarly in Li Paragraphs 348, 363, and statement of motivation in Claim 1.)
Regarding Claim 6: “The method of claim 5, wherein down-sampling comprises average pooling the attention map.” (“A global pooling 2102-1, 2102-2 (also known as average pooling) is a pooling operation that calculates the average value for patches of a feature map … That means, Q and K are reduced to 1D tensors” Zou, Paragraph 434.)
Regarding Claim 7: “The method of claim 1, wherein the attention block is one of a plurality of attention blocks, and … wherein filtering, with the NN-ILF, the current block of video data to generate the filtered current block of video data comprises: … filtering, with a sequence of backbone blocks of the NN-ILF, the current block of video data, wherein each of the backbone blocks is associated with respective one of the plurality of attention blocks.” (Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, backbone blocks can be attention blocks used in the neural network processing. See application of loop filtering to a plurality of blocks based on a plurality of attention blocks in Li Paragraph 332 and Fig. 6: “A CNN may be built upon attention blocks for the in-loop filtering of VVC.”)
Regarding Claim 8: “The method of claim 1, wherein generating the attention map comprises:
generating a query matrix representing input values originating from the current block of video data for which the NN-ILF is identifying relevant context or information from other samples in the current picture; … generating a key matrix representing information relevant to the query matrix; (“Vectors derived from different inputs (e.g., from different patches of an image, or from different images) are then packed together into three different matrices, namely, a query matrix Q 1502, a key matrix (K) 1504 and a value matrix (V) 1506.” Zou, Paragraph 387. See application to loop filtering in Claim 1.)
and generating the attention map based on the query matrix and the key matrix.” (“In the self-attention layer, the input vector is first transformed into three different vectors: the query vector q, the key vector k, and the value vector v with dimension dq=dk=dy=dmodel· Vectors derived from different inputs (e.g., from different patches of an image, or from different images) are then packed together into three different matrices, namely, a query matrix Q 1502, a key matrix (K) 1504 and a value matrix (V) 1506” Zou, Paragraph 387. See application to loop filtering in Claim 1.)
Regarding Claim 9: “The method of claim 1, further comprising decoding the current picture or a subsequent picture based on the filtered current block or encoding the current picture or a subsequent picture based on the filtered current block.” (“The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and/or decoding of audio, image and/or video data or assisting in coding and/or decoding carried out by the controller,” where video data comprises a sequence of pictures including current and subsequent. Zou, Paragraph 225.)
Regarding Claim 10: “The method of claim 1, further comprising inter-prediction encoding or decoding a subsequent block based on the filtered current block of video data.” (“In temporal prediction, the sources of prediction are previously decoded pictures (a.k.a. reference pictures) … Inter-layer or inter-view prediction may be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively.” Zou, Paragraph 243.)
Claim 11: “A device for processing video data,” is rejected for reasons stated for Claim 1, and because prior art teaches: “one or more memories configured to store the video data; and processing circuitry coupled to the one or more memories, wherein the processing circuitry is configured to: …” (“The apparatus 50 may comprise a controller 56, a processor or a processor circuitry for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which in embodiments of the examples described herein may store both data in the form of an image, audio data and video data, and/or may also store instructions for implementation on the controller 56.” Zou, Paragraph 225.)
Claims 12-19 are rejected for reasons stated for Claims 2-8, 10 respectively, in view of the Claim 11 rejection.
Claim 20 “One or more computer-readable storage media storing instructions thereon that when executed cause one or more processors to: …” is rejected for reasons stated for Claim 1, and because prior art teaches: “The apparatus 50 may comprise a controller 56, a processor or a processor circuitry for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which in embodiments of the examples described herein may store both data in the form of an image, audio data and video data, and/or may also store instructions for implementation on the controller 56.” Zou, Paragraph 225.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 11792412 to Rusanovskyy (Rusanovskyy") relevant for using a neural network processor for generating adaptive loop filter data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MIKHAIL ITSKOVICH whose telephone number is (571)270-7940. The examiner can normally be reached Mon. - Thu. 9am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joseph Ustaris can be reached at (571)272-7383. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MIKHAIL ITSKOVICH/Primary Examiner, Art Unit 2483
