DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on 03/17/2026 have been fully considered but they are not persuasive.
Applicant argues: “For example, Janedula does not teach or suggest defining a specific data structure (TUG_ Header) that manages shared transform vectors for a group of units.”
Examiner notes that Janedula teaches placing this information in a parameter set, which is a name used for a header in the modern video coding standards. Cho explicitly teaches a header for the block group. This is cumulative evidence that a header data structure was well known for coding these types of variables. See reasons for rejection for the specific details below.
Applicant further argues that prior art does not teach the newly amended language.
Examiner notes the updated reasons for rejection below that address newly amended claim language.
Applicant argues: “In particular, the cited references do not teach including transform vector size information (basis_ vector size idx) in the transform unit group header to dynamically signal based on the TU sizes of the feature maps.”
Examiner notes that the claims are not directed to dynamical signaling or limit the claim to a particular dynamic determination step. See updated reasons for rejection below.
Applicant may wish to elaborate on the steps embodying the dynamic signaling.
Response to Amendment
Examiner has previously withdrawn the rejection of Claim 18, under 35 U.S.C. 101, in view of the amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 8-13, 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over US 20190042923 to Janedula (“Janedula”) in view of US 20220210435 to Kang (“Kang”) and in view of US 20200074223 to Cho (“Cho”).
Regarding Claim 1: “An encoding method, comprising:
generating multiple feature maps (“convolutional neural networks (CNNs). In CNNs, each layer performs 3D convolution on an input feature map, … each input feature map (IFM) in a set of input feature maps 1502 (IFM 1 through IFM N)” Janedula, Paragraphs 143, 145, and Fig. 15A.)
using an input image; (“As shown in FIG. 28A, an exemplary CNN used to model image processing can receive input 2802 describing the red, green, and blue (RGB) components of an input image” Janedula, Paragraphs 199, 189, 193.)
transforming the feature maps using a transform vector; and (Note that a convolution neural network can operate by element-wise multiplication of vectors or matrices: “a Winograd Processing Element (WPE), which performs a Winograd transform on the input and weight tensor, an element-wise multiplication between the transformed inputs and weights, and a vector accumulate operation that adds the output of the multiplication” Janedula, Paragraphs 149, 189-190.)
generating a bitstream by encoding at least any one of the feature maps, a transform coefficient of the feature maps, or the transform vector, or a combination thereof, (For a processed image, “The broadcasting [a bitstream] of transformed inputs amortizes the cost of the transform with negligible broadcast costs.” Janedula, Paragraphs 145-146. “In some embodiments, graphics processor 300 includes a video codec engine 306 to encode, decode, or transcode media to, from, or between one or more media encoding formats, including, but not limited to Moving Picture Experts Group (MPEG) formats …,” where the encoded media (a feature map) is known as a bitstream. See Janedula, Paragraph 54. Further, “In some embodiments, commands for the media pipeline state 940 include data to configure the media pipeline elements that will be used to process the media objects. This includes data to configure the video decode and video encode logic within the media” where transform coefficients and vector are data to configure media pipeline. See Janedula, Paragraph 125.)
wherein the transforming the feature map includes forming a transform unit group including one or more transform units,” (As described in the original dependent Claim 6, “the transform unit corresponds to a sub-feature map of the feature map.” Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, a transform unit in this case can correspond to a partial feature map of Claims 4 and 5.
Prior art teaches transformation of partial feature maps by a CNN: “In 3D convolution, each input feature map (IFM) in a set of input feature maps 1502 (IFM 1 through IFM N) is convolved 1504 with a corresponding convolution filter (Kl) (e.g., Kl, IFMl through Kl, IFMn). The partial results [generated differential feature maps] 1506 are summed to create a final output feature map (OFM) 1508.” Janedula, Paragraph 145.)
wherein the transform vector is set based on the transform unit group, … wherein the transforming for the transform units included in the transform unit group is performed based on a shared transform vector corresponding to the transform unit group, and (“each input feature map (IFM) in a set of input feature maps 1502 (IFM 1 through IFM N) is convolved … This convolution can be repeated [shared] over the whole [group] input feature map to compute the full output feature map. … For example, a 1 x3 [shared vector] kernel 1511 and a 1x6 input feature map patch can be transformed via a weight transform unit 1513 and an input transform unit 1514.” Janedula, Paragraphs 145-146.)
wherein all transform units in the transform unit group are transformed using the shared transform vector, and” (“This convolution can be repeated [shared] over the whole [group] input feature map to compute the full output feature map. … For example, a 1 x3 [shared transform vector] kernel 1511 and a 1x6 input feature map patch can be transformed via a weight transform unit 1513 and an input transform unit 1514.” Janedula, Paragraphs 145-146.)
Where Janedula does not explicitly state that the media encoding formats such as MPEG encode feature maps, coefficients, or vectors in “a bitstream” at the earliest priority date it would have been understood that a transformed image data is image data to be encoded by the image and video algorithms, and that coefficients and vectors also require to be encoded where they are required for decoding or reverse transformation. See, Janedula Paragraphs 54 and 125.
Kang teaches that these features are indeed a part of the MPEG, HEVC, VVC, and VCM codecs in particular, in the context of encoding video for machine vision: “The VCM encoder multiplexes and transmits bitstreams respectively generated by encoding the features and the video. … encoding the reference feature map, and encoding a residual feature map of each of the remaining frames.” Kang, Paragraphs 7, 13. Cumulatively note encoding of parameters such as coefficients and vectors used by the neural network in Kang, Paragraphs 100.
Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to supplement the teachings of Janedula to encode feature maps and neural network parameters such as coefficients and vectors as taught in Kang, in order to encode the “feature map by using an existing video codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC),” Kang, Paragraph 53.
Finally, in reviewing the present application, there does not seem to be objective evidence that the claim limitations are particularly directed to: addressing a particular problem which was recognized but unsolved in the art, producing unexpected results at the level of the ordinary skill in the art, or any other objective indicators of non-obviousness.
Janedula and Kang do not discuss particulars of transmission and do not teach: “wherein a transform unit group header (TUG header) is transmitted in the bitstream for the transform unit group, … wherein the transform unit group header includes information related to the shared transform vector.” However, Janedula teaches encoding video and configuration data for the decoder using know video coding standards in Paragraphs 54 and 125, and it is conventional under video coding standards to designate a coded data structure and groups of likely coded data structures with shared identifications and parameters in a shared header (also known as a parameter set).
Further, Cho teaches the above claim feature in the context of encoding compressed feature maps under video coding standards, for example “a plurality of pixels included in a feature map compressed [transformed] according to a compression mode in unit blocks … the electronic device 100 may determine a block group [including transform units of the blocks] … add information on the compression mode used for compression and information on the header group to the header.” Cho, Paragraphs 155-156. In the case of Janedula (above) the compression mode and information is related to transform vector used in the compression and required for decompression.
Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to supplement the teachings of Janedula and Kang, such as using transform vectors for feature map compression, to code compression information in a header “wherein a transform unit group header is transmitted for the transform unit group, wherein the transform unit group header includes information related to the shared” compression mode as taught in Cho, in order to store coding information about the transformed / compressed groups of pixels included in a feature map for transmission and decoding. See Cho, Paragraphs 12 and 155.
Finally, in reviewing the present application, there does not seem to be objective evidence that the claim limitations are particularly directed to: addressing a particular problem which was recognized but unsolved in the art, producing unexpected results at the level of the ordinary skill in the art, or any other objective indicators of non-obviousness.
“wherein the transform unit group header includes a transform unit group index (TUG idx), transform vector size information (basis vector size idx), and a number of transform vectors (num of basis vector), and” (As noted above, Cho teaches “the electronic device 100 may determine a block group [including transform units of the blocks] … add information on the compression mode used for compression and information on the header group to the header.” Cho, Paragraphs 155-156. As noted in Janedula (above) the information on the compression mode can include “a value for the second kernel [transform vector] size, … to perform a Winograd convolution operation based on the value for the second kernel size: Paragraph 226. The information can also include: “multiple sub-kernels [vector] having a second kernel size,” for example, “The Winograd compute block 2100 can accept as input 2102 a 1x6 transformed kernel” indicating that 6 sub-kernels/vectors are used in the group. See Janedula, Paragraphs 228, 162, and Fig. 21. In this example, the information on the used compression mode includes kernel/vector size and number, and in view of Cho, it would have been known and obvious to include such information in the header group. See statement of motivation above.)
“wherein the transform vector size information in the transform unit group header is signaled based on transform unit sizes of feature maps included in the transform unit group.” (First, note that the claim uses the “the transform vector size information” but is not limited to performing the steps of determining it. See treatment of product by process limitations above. Cumulatively, prior art teaches: “multiple sub-kernels having a second kernel size; transforming a portion of an input feature map and the multiple sub-kernels based on a Winograd transform, the Winograd transform associated with the second kernel size;” thus the transform vector size information is based on transform unit sizes of feature maps being transformed. This appears to be a direct functional connection. See Janedula, Paragraphs 228, 162, and Fig. 21.)
Regarding Claim 2: “The encoding method of claim 1, wherein generating the bitstream includes packing at least any one of the feature maps, the transform coefficient of the feature maps, or the transform vector, or the combination thereof.” (As noted in Claim 1: “graphics processor 300 includes a video codec engine 306 to encode [pack], decode, or transcode media to, from, or between one or more media encoding formats, including, but not limited to Moving Picture Experts Group (MPEG) formats …,” where the encoded media (a feature map) is known as a bitstream. See Janedula, Paragraph 54. Further, “In some embodiments, commands for the media pipeline state 940 include data to configure the media pipeline elements that will be used to process the media objects. This includes data to configure the video decode and video encode logic within the media” where transform coefficients and vector are data to configure media pipeline. See Janedula, Paragraph 125.)
Regarding Claim 3: “The encoding method of claim 1, wherein generating the multiple feature maps comprises using an artificial neural network structure configured with multiple layers.” (“The nodes in the CNN input layer are organized into a set of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network.” Janedula, Paragraphs 193, 199.)
Regarding Claim 4: “The encoding method of claim 3, wherein generating the multiple feature maps comprises extracting a pan of feature maps corresponding to the multiple layers.” (“In CNNs, each layer performs 3D convolution on an input feature map … In 3D convolution, each input feature map (IFM) in a set of input feature maps 1502 (IFM 1 through IFM N)” thus multiple feature maps are processed by corresponding multiple CNN layers. Janedula, Paragraphs 143, 145.)
Regarding Claim 5: “The encoding method of claim 1, wherein generating the multiple feature maps comprises generating a differential feature map between a predicted feature map and an original feature map.” (“In 3D convolution, each input feature map (IFM) in a set of input feature maps 1502 (IFM 1 through IFM N) is convolved 1504 with a corresponding convolution filter (Kl) (e.g., Kl, IFMl through Kl, IFMn). The partial results [generated differential feature maps] 1506 are summed to create a final output feature map (OFM) 1508.” Janedula, Paragraph 145.)
Regarding Claim 8: “The encoding method of claim 1, wherein transforming the feature map includes down-sampling or up-sampling the transform unit when a size of the transform vector differs from a size of the transform unit.” (“For example, a 1 x3 [vector] kernel 1511 and a 1x6 input feature map patch can be transformed via a weight transform unit 1513 and an input transform unit 1514. … An output transform unit 1516 transforms the 1x6 intermediate output matrix into a 1 x4 output 1518” thus the vector and the feature map transform unit differ in size and the output feature map is downsized from 1x6 to 1x4. Janedula, Paragraph 146.)
Claims 9-13, 16, “A decoding method” are rejected for reasons stated for Claims 1-5, 8 respectively, because the decoding steps of the former are “inversely” performing the encoding steps of the latter, and because prior art teaches “to encode, decode, or transcode media” using the same methodology and computing resources in Janedula, Paragraph 54.)
Regarding Claim 17: “The decoding method of claim 12, wherein reconstructing the multiple feature maps comprises
reconstructing other feature maps using a result of performing a convolution operation on the inversely transformed feature map (“the logic 1700 can perform an inverse transform operation 1714 in which an inverse Winograd transform is performed on a summed output feature map Yn …” of the type created by the encoding claims above. See Janedula, Paragraph 150.)
and a residual feature map.” (Janedula does not talk about a residual feature map, however this is a common feature of the MPEG and H.264 encoding processes, as noted in Kang, Paragraph 13: “encoding the reference feature map, and encoding a residual feature map.” In a corresponding manner, “The VCM decoder obtains a multiplexed bitstream to thus obtain a bitstream corresponding to a residual feature map and a bitstream corresponding to an input image. The VCM decoder decodes the residual feature map from the bitstream. The VCM decoder generates a reconstructed residual feature map by applying a feature map inverse conversion to the residual feature map.” Kang, Paragraph 63. See statement of motivation in Claim 1.)
Claim 18, “A non-transitory computer-readable recording medium for storing a bitstream,” is rejected for reasons stated for Claims 1 and 8 which respectively encode and decode the content of Claim 18, and because prior art teaches: “one or more machine-readable storage media and machine-readable communication media. Thus, the storage devices of a given electronic device typically store code and/or data for execution on the set of one or more processors of that electronic device.” Janedula, Paragraph 232.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
G. J. Sullivan, J. -R. Ohm, W. -J. Han and T. Wiegand, "Overview of the High Efficiency Video Coding (HEVC) Standard," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649-1668, Dec. 2012, doi: 10.1109/TCSVT.2012.2221191 (“Sullivan”) is relevant for teaching that features of the HEVC standard are set up to code video data in the clamed manner.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MIKHAIL ITSKOVICH whose telephone number is (571)270-7940. The examiner can normally be reached Mon. - Thu. 9am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joseph Ustaris can be reached at (571)272-7383. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MIKHAIL ITSKOVICH/Primary Examiner, Art Unit 2483
Prosecution Insights