DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2,9-10,14,18-19,24 rejected under 35 U.S.C. 103 as being unpatentable over Ding; Ding et al. (US 20230316588 A1) in view of Rusanovskyy; Dmytro et al. (US 20200288158 A1) in view of Bourdev; Lubomir et al. (US 20230018461 A1)
Regarding claim 1, Ding teaches,
A data processing method (title, “encoder tuning with multi model selection in neural image compression”) comprising:
receiving, by a first processor, (¶136-138 and fig. 15, “encoding device (1510)” depicted in fig. 15) input data; (¶136-138 and fig. 15, “encoding device (1510) receives an input image”)
selecting, by the first processor, an encoder (¶136-138 and fig. 15, “encoding device (1510) can select one of the encoders in the encoder set (1520)”) to encode the input data among a plurality of encoders based on the input data; (¶136-138 and fig. 15, “encoding device (1510) can choose an encoder” such that the chosen encoder is used to “compress the input image into a coded bitstream”)
generating, by the first processor, (¶136-138 and fig. 15, “chosen encoder” from the encoder set (1520) of the encoding device (1510) depicted In fig. 15) encoded data (¶136-138, compressed “input image into a coded bitstream”) and identification data (¶136-138 and fig. 15, “index indicative of the chosen encoder can be signaled” in the coded bitstream) that identifies a decoder (¶136-138, index indicative of the chosen encoder used to “determine a decoder”) to decode the encoded data (¶136-138,”determined decoder” is then used to “decode the coded bitstream”) among a plurality of decoders. (¶136-138, index indicative of the chosen encoder used to “determine a decoder from the decoder set (1570)”)
But does not explicitly teach,
obtaining, by the first processor, a vector value by encoding the input data using the selected encoder; and
encoded data comprising the vector value, to decode the vector value, wherein the identification data comprises a value for identifying an encoder-decoder pair that is simultaneously trained among a plurality of encoder-decoder pairs.
However, Rusanovskyy teaches,
obtaining, by the first processor, (¶150, “video encoder 200”) a vector value by encoding the input data (¶150, derive motion vector of each 4x4 sub-block when coder calculates “motion vector of the center sample of each sub-block”) using the selected encoder; (¶150 and 97-98, “mode selection unit 202” selects encoding parameters in accordance with “prediction modes”) and
generating encoded data comprising the vector value, (¶150, “mode selection unit 202 of video encoder 200” may calculate the “motion vector” of each 4x4 sub-block) and identification data (¶102 and 88-89, “entropy encoding unit 220” of encoder 200 encodes a “syntax element indicating the filter index value”) that identifies a decoder to decode the vector value (¶102, 88-89, and 150, syntax element “syntax element signaled for the video data” indicating the filter index value for the video data used to “calculate the motion vector”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy that indicates indexes for video data. This form of signaling can reduce energy consumption by limiting the data signaled in a bitstream.
Bourdev teaches additionally,
wherein the identification data comprises a value for identifying an encoder-decoder pair (¶32, “cloud service system 130 may identify one or more decoder filters associated with bitstream compatibility attributes that indicate the decoder filter can process bitstreams generated by the particular encoder filter”) that is simultaneously trained among a plurality of encoder-decoder pairs. (¶32 and 21, decoder filter can process bitstreams generated by the particular encoder filter “pair of compatible filters and the decoder filter may be referred to as a paired (or compatible) decoder filter for the encoder filter (or vice versa)”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev which trains encoder and decoder filter pairs in conjunction with each other. This allows for reconstructing information in a way to achieve certain functions like high-resolution or obscuring objects in a video.
Regarding claim 2, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
Ding teaches additionally,
identification data (¶136-138, “index indicative of the chosen encoder can be signaled” in the bitstream) comprises a flag value or index value (¶136-138, “index indicative of the chosen”) for identifying the decoder among the plurality of decoders (¶136-138, “based on the index, the decoding device (1560) can determine a decoder from the decoder set (1570)”) corresponding to the plurality of encoders. (¶136-138 and fig. 15, decoder extracted “index from the coded bitstream” associated with index indicative of the chosen encoder in the “coded bitstream” from encoding device (1510) depicted in fig. 15)
Regarding claim 9, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
Rusanovskyy teaches additionally,
input data (¶93,150, and fig. 8, “video data for a current block” that the video encoder 200 “receives for encoding”) comprises pixel values of pixels included in a local region of an image. (¶208,150,93, and fig. 8, process for each “set of samples for the pixel” with values for the pixel of “4x4 sub-block” 570 depicted in fig. 8)
Regarding claim 10, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
Ding teaches additionally,
input data comprises at least one of image data, video data, audio data, or any combination thereof. (¶136-138 and fig. 15, encoding device (1510) “receives an input image” compressed into a coded bitstream)
Regarding claim 14, it is the electronic device claim of method claim 1.
Ding teaches additionally,
An electronic device for performing a data processing method, (title, “encoder tuning with multi model selection in neural image compression”) the electronic device comprising:
a first processor; (¶143,136-138,175, and fig. 17, “processing circuitry” performing process by software depicted in fig. 17 associated with an “encoding device” by processor) and
a memory configured to store instructions (¶143 and 175, software embodied in “computer-readable media” associated with user-accessible mass storage) to be executed by the first processor, (¶175, “processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software”)
wherein when the instructions are executed by the first processor, (¶143 and 136-138, “processing circuitry executes the software instructions, the processing circuitry performs the process”)
Refer to method claim 1 to teach the limitations of electronic device claim 14.
Regarding claim 18, dependent on claim 14, it is the electronic device of method claim 4, dependent on claim 3. Refer to rejection of claim 4 to teach the limitations of claim 18.
Regarding claim 19, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 18,
Ding teaches additionally,
the plurality of encoders (¶136-138 and fig. 15, “encoding set (1520) that includes a plurality of encoders”) comprise a first encoder and a second encoder, (¶136-138 and fig. 15, encoding set (1520) includes a plurality of encoders such as “encoder 1” and “encoder 2” as depicted in fig. 15)
the plurality of decoders (¶136-138 and fig. 15, “decoding set (1570) that includes a plurality of decoders”) comprise a first decoder paired with the first encoder (¶136-138 and fig. 15, decoding set (1570) including “decoder 1” that corresponds to encoder 1 as depicted in fig. 15) and a second decoder paired with the second encoder, (¶136-138 and fig. 15, decoding set (1570) including “decoder 2” that corresponds to encoder 2 as depicted in fig. 15) and
a first pair of the first encoder and the first decoder (¶136-138 and fig. 15, “decoder 1 corresponds to encoder 1”) and a second pair of the second encoder and the second decoder (¶136-138 and fig. 15, “decoder 2 corresponds to encoder 2”) are trained based on different loss functions. (¶136-141 and fig. 15, “decoder 1 corresponds to encoder 1” of a first pretrained NIC framework and “decoder 2 corresponds to encoder 2” of a second pretrained NIC framework that can achieve a least loss “(e.g., a least rate loss, a least distortion loss, a least rate distortion loss)” with online training based on the selected tuning)
Regarding claim 24, it is the non-transitory computer-readable recording medium claim of method claim 1.
Ding teaches additionally,
A non-transitory computer-readable recording medium storing instructions (¶143 and 175, software embodied in “computer-readable media” associated with user-accessible mass storage of “non-transitory nature”) that, when executed by a processor, (¶175, “processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software”) cause the processor to perform operations comprising, (¶143 and 136-138, “processing circuitry executes the software instructions, the processing circuitry performs the process”)
Refer to method claim 1 to teach the limitations of non-transitory computer-readable recording medium claim 24.
Claim(s) 3-6,8,15-16 rejected under 35 U.S.C. 103 as being unpatentable over Ding; Ding et al. (US 20230316588 A1) in view of Rusanovskyy; Dmytro et al. (US 20200288158 A1) in view of Bourdev; Lubomir et al. (US 20230018461 A1) in view of KIM; Sunyeon et al. (US 20110317930 A1)
Regarding claim 3, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
Ding teaches additionally,
receiving, by a second processor, (¶136-138 and fig. 15, “decoding device (1560)”) the encoded data and the identification data; (¶136-138 and fig. 15, “coded bitstream can be transmitted to the decoding device (1560)” which includes compressed “input image into a bitstream” and “index indicative of the chosen encoder” signaled in the coded bitstream)
selecting, by the second processor, (¶136-138 and fig. 15, “decoding device (1560)”) the decoder to decode the value, (¶136-138, “determined decoder” used to “decode the coded bitstream”) among the plurality of decoders, (¶136-138, determined decoder “from the decoder set (1570)”) based on the identification data (¶136-138, “index” extracted from the coded bitstream used to “determine a decoder from the decoder set (1570)”) and
obtaining, by the second processor, (¶136-138 and fig. 15, “decoding device (1560)”) reconstructed data corresponding to the input data (¶136-138, “determined decoder is then used to decode the coded bitstream to generate a reconstructed image”) by performing decoding on the value comprised in the encoded data (¶137-138, determined “decoder corresponds to the chosen encoder at the encoding device (1510)”) using the selected decoder. (¶136-138, “based on the index, the decoding device (1560) can determine a decoder from the decoder set (1570)”)
But does not explicitly teach,
decode the vector value
using a demultiplexer that determines which of the plurality of decoders to transmit the vector value to based the identification data;
performing decoding on the vector value
Rusanovskyy teaches additionally,
selecting a decoder to decode the vector value (¶88-89, video decoder 300 may “determine a filter index value” based on “a syntax element signaled for the video data”
performing decoding on the vector value (¶150, “prediction processing unit 304 of video decoder 300” may calculate the “motion vector” of 4x4 sub-block)
Kim teaches additionally,
selecting based on the identification data (¶92 and fig. 3, “differentiate the encoded motion vector prediction mode and the encoded differential vector”) using a demultiplexer that determines which of the plurality of decoders to transmit the vector value to based the identification data; (¶92 and fig. 3, “demultiplexer is used to differentiate the encoded motion vector prediction mode and the encoded differential vector” such that “ motion vector prediction mode encoded in the upper unit is inputted to prediction mode decoder 310, and the differential vector encoded for each block is inputted to the motion vector prediction mode determiner and differential vector decoder 320” as depicted in fig. 3)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev with the demultiplexer of Kim which directs vector information from a bitstream to particular decoders of a decoding apparatus. This type of exacting selection improves coding efficiency and solves problems of bit quantity due to encoding additional information.
Regarding claim 4, Ding with Rusanovskyy with Bourdev with Kim teaches the limitations of claim 3,
Ding teaches additionally,
each of the plurality of decoders (¶136-138 and fig. 15, determined “decoder from the decoder set (1570)”) has a relationship of a pair (¶136-138, “decoder corresponds to the chosen encoder”) with one of the plurality of encoders, (¶136-138 and fig. 15, “chosen encoder” selected from the encoders in the encoder set (1520)” depicted in fig. 15) and
the number of the plurality of decoders is equal to or less than the number of the plurality of encoders. (¶138 and fig. 15, “encoders in encoder set (1520) can be pretrained NIC encoders of pretrained NIC frameworks and the decoders in the decoder set (1570) can be pretrained NIC decoders of the pretrained NIC frameworks” the same network structure such that encoder set (1520) has encoders up to “encoder n” and decoder set (1570) has decoders up to “decoder n” as depicted in fig. 15)
Regarding claim 5, Ding with Rusanovskyy with Bourdev with Kim teaches the limitations of claim 3,
Ding teaches additionally,
a first pair of the first encoder and the first decoder (¶136-142 and fig. 15-16, “encoder 1 and decoder 1 are pretrained NIC encoder and pretrained NIC decoder of a first pretrained NIC framework”) and a second pair of a second encoder and a second decoder (¶136-142 and fig. 15-16, encoder 2 and decoder 2 are “pretrained NIC encoder and pretrained NIC decoder” of a “second pretrained NIC framework (e.g., a second NIC model)”) are trained based on different loss functions. (¶141, first and second “pretrained NIC framework” that can achieve a least loss “(e.g., a least rate loss, a least distortion loss, a least rate distortion loss)” with online training based on the selected tuning)
Regarding claim 6, Ding with Rusanovskyy with Bourdev with Kim teaches the limitations of claim 3,
Ding teaches additionally,
encoded data and the identification data are transmitted (¶136-138 and fig. 15, “compress the input image into a coded bitstream” and “index indicative of the chosen encoder” signaled in the coded bitstream ) from the first processor to the second processor, (¶136-138 and fig. 15, “coded bitstream” output from “encoding device (1510)”, depicted in fig. 15, includes compressed “input image” and “index” indicative of the chosen encoder “transmitted to the decoding device (1560)”) or output from the first processor, stored in a memory, and then transmitted to the second processor.
Regarding claim 8, Ding with Rusanovskyy with Bourdev with Kim teaches the limitations of claim 3,
Ding teaches additionally,
the first processor (¶136-143 and fig. 15-16, encoding device such as “encoding device (1610)” depicted in fig. 16 and/or encoding device (1510) depicted in fig. 15) is included in a first electronic device, (¶136-143 and fig. 16, process executed in an “electronic device, such as the encoding device (1610)”) and
the second processor (¶136-138,156, and fig. 15, “decoding device (1560)” depicted in fig. 15) is included in a second electronic device. (¶136-138,156, and fig. 15, process executed in an “electronic device, such as the decoding device (1560)” which appears separate and distinct from an encoding device as depicted in the image coding system (1500) of fig. 15)
Regarding claim 15, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 14,
Ding teaches additionally,
a second processor, (¶156,136-138, and fig. 18, “decoding device” executing software instructions when “processing circuitry executes the software instructions”)
wherein the second processor (¶156, “processing circuitry” executed in an electronic device “executes the software instructions”) is configured to:
The additional limitations of claim 15 are similar to the limitations claimed in method claim 3, dependent on claim 1. Refer to method claim 3 to teach the limitations of electronic device claim 15.
Regarding claim 16, dependent on claim 14, it is the electronic device of method claim 2, dependent on claim 1. Refer to rejection of claim 2 to teach the limitations of claim 16.
Claim(s) 7 rejected under 35 U.S.C. 103 as being unpatentable over Ding; Ding et al. (US 20230316588 A1) in view of Rusanovskyy; Dmytro et al. (US 20200288158 A1) in view of Bourdev; Lubomir et al. (US 20230018461 A1) in view of KIM; Sunyeon et al. (US 20110317930 A1) in view of COBAN; Muhammed Zeyd et al. (US 20220086463 A1)
Regarding claim 7, Ding with Rusanovskyy with Bourdev with Kim teaches the limitations of claim 6,
But does not explicitly teach the additional limitations of claim 7,
However, Coban teaches additionally,
first processor, (¶147,138, and fig. 1, SOC 100 implementing “encoding operations of process 700” implemented as stored instructions executed by “one or more processors”) the second processor, (¶147, and fig. 1, SOC 100 implementing “process 800” corresponding to “decoding operations” implemented as stored instructions executed by “one or more processors”) and the memory (¶61,147,138, and fig. 1, “memory 118” as part of “SOC 100” as depicted in fig. 1) are included in a system-on-chip (SoC). (¶147,61, and fig. 1, SOC 100 associated with “system-on-a-chip (SOC) 100” depicted in fig. 1)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev with the demultiplexer of Kim with the system-on-a-chip of Coban which implements encoding and decoding processes. This allows for operations that can improve coding devices by efficiently balancing resource usage and video image quality.
Claim(s) 11,21 rejected under 35 U.S.C. 103 as being unpatentable over Ding; Ding et al. (US 20230316588 A1) in view of Rusanovskyy; Dmytro et al. (US 20200288158 A1) in view of Bourdev; Lubomir et al. (US 20230018461 A1) in view of Boyce; Jill et al. (US 20210258590 A1)
Regarding claim 11, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
But does not explicitly teach the additional limitations of claim 11,
However, Boyce teaches additionally,
plurality of encoders are connected to a multiplexer, (¶40 and fig. 2, “ descriptions, A and B, each of which is encoded independently of the other description via encoders A and B” which output resultant bitstream that are “multiplexed”) and the multiplexer is configured to control a connection (¶40 and fig. 2, “resultant bitstreams are multiplexed by the multiplexer of multiple description encoder system 201 to generate resultant bitstream 105” depicted in fig. 2) between the plurality of encoders (¶40, “resultant bitstreams are multiplexed by the multiplexer of multiple description encoder system 201”) and a data storage so that the vector value is stored in the encoded data. (¶40, resultant bitstreams from “encoder system 201” are multiplexed transmitted to “memory for eventual decode”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev with the multiplexer of Boyce which generates a bitstream from multiple encoders. This technique provides improvements to coding efficiency and video quality of immersive video by providing scalable coding.
Regarding claim 21, dependent on claim 14, it is the electronic device of method claim 11, dependent on claim 1. Refer to rejection of claim 11 to teach the limitations of claim 21.
Claim(s) 22-23 rejected under 35 U.S.C. 103 as being unpatentable over Ding; Ding et al. (US 20230316588 A1) in view of Rusanovskyy; Dmytro et al. (US 20200288158 A1) in view of Bourdev; Lubomir et al. (US 20230018461 A1) in view of Lefebvre; Frederic et al. (US 20260136032 A1)
Regarding claim 22, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 1,
But does not explicitly teach the additional limitations of claim 22,
However, Lefebvre teaches additionally,
the plurality of encoders comprises convolutional layer. (¶44, “encoder neural networks are composed of multiple layers, such as convolutional layers”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev with the convolutional layers of Lefebvre which compose an encoder neural network with multiple layers. This allows for backward pass and forward passing of learning stages which can update parameters to minimize the cost function.
Regarding claim 23, Ding with Rusanovskyy with Bourdev teaches the limitations of claim 3,
But does not explicitly teach the additional limitations of claim 22,
However, Lefebvre teaches additionally,
Selected decoder is implemented as a neural network model (¶44, “decoder encoder neural networks are composed of multiple layers, such as convolutional layers”) comprising a transposed convolutional layer. (¶85, “deconvolutional layers in the decoder neural network” as part of the encoder neural network “while continuing learning, i.e., updating, the parameters of the convolutional layers in the encoder neural network”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to combine the image compression of Ding with the syntax element of Rusanovskyy with the filter identification of Bourdev with the convolutional layers of Lefebvre which compose an encoder neural network with multiple layers. This allows for backward pass and forward passing of learning stages which can update parameters to minimize the cost function.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIMMY S LEE whose telephone number is (571)270-7322. The examiner can normally be reached Monday thru Friday 10AM-8PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joseph G. Ustaris can be reached at (571) 272-7383. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSEPH G USTARIS/Supervisory Patent Examiner, Art Unit 2483
/JIMMY S LEE/Examiner, Art Unit 2483
Prosecution Insights