DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-3,5-11,13-19 and 41-43 are currently pending in the present application, with claims 1, 9, and 17 being independent.
Response to Amendments / Arguments
Applicant’s arguments, see Pg. 11, filed 11/11/2025, with respect to claim 7 have been fully considered and are persuasive. The objection of claim 7 has been withdrawn.
Applicant’s arguments, see Pg. 11, filed 11/11/2025, with respect to claims 3, 11, and 19 under 35 U.S.C. § 112 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly amended claims.
Applicant’s arguments with respect to claim(s) 1-3, 5-11, and 13-19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 11, and 19 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites “Convert the first division information into second division information in the fisheye format”, and further requires generating blocks based on the second division information. It is unclear what constitutes as “second division information”, yet requires generating blocks based on that information. Claim 3 depends from claim 1 where claim 1 defines “division information” as “division information specifying a number of processor cores to be used to encode stitched tiles”, while claim 3’s introduction of a format-conversion of “first division information” to “second division information” creates ambiguity as to whether the converted information is the core/allocation-based division information of claim 1. Even if one of ordinary skill in the art were to interpret claim 3 as attempting to define a tile layout in a rectangular projection as “first division information”, convert that tile layout into fisheye coordinates as “second division information”, and use the converted information to generate fisheye blocks that map back to the rectangular tiles, the claims still fail to reasonably ascertain what the required “conversion” entails. The claim and originally filed disclosure does not specify whether the conversion preserves the number of tiles, tile location information, tile sizes, or merely converts coordinate representation and how the first division information in rectangular format is converted into second division information in the fisheye format. Paragraph [0034] - [0035] of the originally filed specification merely states that division information may be mapped and/or converted between formats using one or more equations, but does not define the structure, content, or boundaries of the alleged “second division information,” nor does it clarify how it is meaningfully distinct from the first division information. Therefore, it does not define the structure or invariants of the resulting “second division information” and the scope of claim 3 cannot be reasonable ascertain.
The examiner respectfully requests the applicant to clarify the scope of the claimed invention.
Claims 11 and 19 recites substantially similar subject matter as to that of claim 3 and is rejected using substantially similar rationale as to that which was set forth with respect to claim 3.
Claims 3, 11, and 19 will be examined as best understood by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7, 9-11, 13, 15, 17-19, and 41-43 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (US 20190110076), hereinafter referred to as “Lim”, in view of Khan et al. "Software architecture of high efficiency video coding for many-core systems with power-efficient workload balancing." In 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1-6. IEEE, 2014, hereinafter referred to as “Khan”.
Regarding claim 1, Lim discloses an apparatus comprising:
tile generation circuitry (Par. 0063; Splitter 102 splits each picture included in an input moving picture into blocks.) to generate first input tiles from a first image (Fig. 14; Angle 1. Fig. 15; Image A) and second input tiles from a second image (Fig. 14; Angle 2. Fig. 15; Image B), the stitched tiles based on the first input tiles (Fig. 14; blocks/tiles of Angle 1. Fig. 15; blocks/tiles of Image A) and the second input tiles (Fig. 14; blocks/tiles of Angle 2. Fig. 15; blocks/tiles of Image B, the first image from a first camera (Fig. 14; LENS A. Fig. 15; Cameras covering more than one view) and the second image from a second camera (Fig. 14; LENS B. Fig. 15; Cameras covering more than one view. Par. 0159; As illustrated in FIG. 14 and FIG. 15, an image to be input into the encoder may be the result of a stitching process that combines a plurality of images from different cameras),
stitching circuitry to generate the stitched tiles (Par. 0159-160; stitching process that combines a plurality of images from different cameras) based on the first input tiles (Fig. 14; blocks/tiles of Angle 1. Fig. 15; blocks/tiles of Image A) and the second input tiles (Fig. 14; blocks/tiles of Angle 2. Fig. 15; blocks/tiles of Image B)
encoding circuitry (Par. 0148-0153 Embodiment 2; adaptive video encoding tools can be adaptive based on the image conversion or image stitching processes used to process the images prior to the encoder. Par. 0154-0156; encoding process).
Lim does not disclose based on division information specifying a number of processor cores to be used to encode stitched tiles, and including the processor cores to encode stitched tiles in parallel, respective ones of the stitched tile to be encoded with corresponding ones of the processor cores.
In the same art of encoding tiles, Khan discloses based on division information specifying a number of processor cores to be used to encode stitched tiles (Fig. 6 and Section IV, right column; While encoding, the input video is converted into tiles and forwarded to the respective cores. Cores process each tile individually…Fig. 7; Input: Maximum number of cores K…Output: Total tiles Ttot and initial θinit per tile…), and , and including the processor cores to encode stitched tiles in parallel (Section III.B; Par. 1; tiles can be processed via independent encoding threads…utilizing HEVC parallelism at tile-level…), respective ones of the stitched tile to be encoded with corresponding ones of the processor cores (Fig. 6-7 and Section IV, right column; While encoding, the input video is converted into tiles and forwarded to the respective cores. Cores process each tile individually…).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Khan’s tile-based encoding architecture in which tiles are generated and distributed based on available processor cores for parallel execution into Lim’s tiled stitching and encoding pipeline. The motivation lies in the advantage of improved computational efficiency, reduce encoding latency, and better utilize multi-core processing resources when handling stitched tile outputs (Khan Abstract; By balancing the workload of each video tile mapped to a particular core, the total power consumption of a system is reduced (through dynamically scaling the operating frequency) under a given frame-rate constraint). Such integration would have predictably enabled scalable performance as the number of tiles increases by applying known parallelization techniques to a compatible tiled architecture.
Regarding claim 2, Lim in view of Khan discloses the apparatus of claim 1, and further discloses wherein the first image and the second image are in a fisheye format (Lim Fig. 14, Fig. 17, and Par. 0149-0151; the encoder, the decoder, and the like according to this embodiment can be used in the encoding and decoding of an image captured with a non rectilinear (e.g., a fisheye) lens camera…a 360-degree image is originally captured by multiple cameras and the images captured from the multiple cameras) and the first input tiles and the second input tiles are in a rectangular format (Lim Fig. 15 and Par. 0186; As illustrated in FIG. 13, the captured image may be distorted due to the characteristics of the lens used during the capturing of the image. An image correction process was used to rectify the captured image to rectilinear.
Lim and Khan are combined for the reason set forth above with respect to claim 1.
Regarding claim 3, Lim in view of Khan discloses the apparatus of claim 1. Lim further discloses wherein the division information is first division information that specifies at least one of tile location information or tile size information in rectangular format (Par. 0063; Splitter 102 splits each picture included in an input moving picture into blocks, and outputs each block to subtractor 104. For example, splitter 102 first splits a picture into blocks of a fixed size (for example, 128×128) …Par. 0105; performed at another level…tile level), and the tile generation circuitry to:
convert the first division information (Par. 0163; the encoder can wrap pixels in an image to correct the distortion or reverse the correction done to make an image rectilinear. In other words, the encoder performs an image correction process (i.e., a wrapping process) on distorted blocks in an image to be encoded) into second division information in the fisheye format (Par. 0163; The encoder may also return the prediction block, which is a predicted block, to its original distorted state before undergoing the image correction process);
generate blocks (Par. 0163; the encoder can perform a block prediction to derive a block of prediction samples based on the wrapped image pixels) of the first image in the fisheye format based on the second division information (Fig. 14; Lens A (Angle 1));
generate blocks (Par. 0163; the encoder can perform a block prediction to derive a block of prediction samples based on the wrapped image pixels) of the second image in the fisheye format based on the second division information (Fig. 14; Lens B (Angle 2));
map the blocks (Fig. 13 and Par. 0186; An image correction process was used to rectify the captured image to rectilinear. The parsed parameters include such parameters to identify or describe the image correction process used. Examples of parameters used in the image correction process include parameters configuring a mapping table to map input image pixels to the intended output pixel values of the image correction process. These parameters may include weight parameters for one or more interpolation process or/and position parameters identifying the locations of the input and output pixels in a picture. In one possible implementation example of the image correction process, the mapping table for the image correction process may be used for all the pixels in the corrected image) of the first image in the fisheye format to corresponding ones of the first input tiles in the rectangular format (Fig. 13-15 and Par. 0149-0151; the encoder, the decoder, and the like according to this embodiment can be used in the encoding and decoding of an image captured with a non rectilinear (e.g., a fisheye) lens camera…a 360-degree image is originally captured by multiple cameras and the images captured from the multiple cameras); and
map blocks (Fig. 13 and Par. 0186; An image correction process was used to rectify the captured image to rectilinear. The parsed parameters include such parameters to identify or describe the image correction process used. Examples of parameters used in the image correction process include parameters configuring a mapping table to map input image pixels to the intended output pixel values of the image correction process. These parameters may include weight parameters for one or more interpolation process or/and position parameters identifying the locations of the input and output pixels in a picture. In one possible implementation example of the image correction process, the mapping table for the image correction process may be used for all the pixels in the corrected image) of the second image in the fisheye format to corresponding ones of the second input tiles in the rectangular format (Fig. 13-15 and Par. 0149-0151; the encoder, the decoder, and the like according to this embodiment can be used in the encoding and decoding of an image captured with a non rectilinear (e.g., a fisheye) lens camera…a 360-degree image is originally captured by multiple cameras and the images captured from the multiple cameras).
Lim and Khan are combined for the reason set forth above with respect to claim 1.
Regarding claim 5, Lim in view of Khan discloses the apparatus of claim 1, and further discloses wherein the tile generation circuitry is to generate the first input tiles and the second input tiles based on parameters of the first camera and the second camera (Lim Fig. 14 and Fig. 15).
Lim and Khan are combined for the reason set forth above with respect to claim 1.
Regarding claim 7, Lim in view of Khan discloses the apparatus of claim 1, and further discloses
including at least one memory (Par. 0061; processor and memory), wherein the tile generation circuitry is to:
(i) store ones of the first input tiles in respective first isolated memory blocks (Par. 0087; block memory 118 is storage for storing blocks in a picture to be encoded (hereinafter referred to as a current picture) for reference in intra prediction) of the at least one memory (Par. 0095; Frame memory 122 stores reconstructed blocks filtered by loop filter 120. Par. 0061; processor and memory) and
(ii) store ones of the second input tiles in respective second isolated memory blocks (Par. 0087; block memory 118 is storage for storing blocks in a picture to be encoded (hereinafter referred to as a current picture) for reference in intra prediction) of the at least one memory (Par. 0095; Frame memory 122 stores reconstructed blocks filtered by loop filter 120. Par. 0061; processor and memory) , and the stitching circuitry is to
(i) replace the ones of the first input tiles in the respective first isolated memory blocks with corresponding first ones of the stitched tiles (Par. 176-0181, 0292 and Fig. 23; reconstruct the encoded image to generate a reconstructed image; and store an image obtained by stitching the reconstructed image and with the second image into the memory as a reference frame to be used in an inter prediction process), and
(ii) replace the ones of the second input tiles in the respective second isolated memory blocks with corresponding second ones of the stitched tiles (Par. 176-0181, 0292 and Fig. 23; reconstruct the encoded image to generate a reconstructed image; and store an image obtained by stitching the reconstructed image and with the second image into the memory as a reference frame to be used in an inter prediction process).
Lim and Khan are combined for the reason set forth above with respect to claim 1.
Regarding claim 9, claim 9 is the CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media) of apparatus claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 10, claim 10 has similar limitations as of claim 2, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 2.
Regarding claim 11, claim 11 has similar limitations as of claim 3, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 3.
Regarding claim 13, claim 13 has similar limitations as of claim 5, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 5.
Regarding claim 15, claim 15 has similar limitations as of claim 7, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 7.
Regarding claim 17, claim 17 is the system claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer program, computer-readable recording media) of apparatus claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Regarding claim 18, claim 18 has similar limitations as of claim 2, except it is a system claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer program, computer-readable recording media), therefore it is rejected under the same rationale as claim 2.
Regarding claim 19, claim 19 has similar limitations as of claim 3, except it is a system claim (Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer program, computer-readable recording media), therefore it is rejected under the same rationale as claim 3.
Regarding claim 41, Lim in view of Khan discloses the apparatus of claim 1, but Lim does not disclose wherein the division information includes at least one of tile location information and tile size information based on (i) the number of processor cores, (ii) bandwidth associated with the processor cores, and (iii) input size requirements associated with the processor cores.
In the same art of encoding tiles, Khan discloses wherein the division information includes at least one of tile location information and tile size information (Fig. 2 and Section III.A; Each GOP contains M video frames and there are K tiles (T) in each frame, where K ≥ 1. A frame can also be divided into independent slices. In HEVC, each tile is divided into a Coding Tree Units (CTU) of size 64×64 pixels or smaller. Fig. 7) based on (i) the number of processor cores (Fig. 7; Maximum number of cores K), (ii) bandwidth associated with the processor cores (Fig. 6), and (iii) input size requirements associated with the processor cores (Fig. 7; Input: Maximum number of cores K; Maximum frequency of a core fmax; Image dimensions W×H; Quantization Parameter QP; Frame rate in frames per second fp;).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to apply Khan’s resource-aware division techniques to Lim’s tiled image processing system. The motivation lies in the advantage of improving computational efficiency and load balancing in multi-core encoding systems by adapting task partitioning based on available processor cores and input characteristics, resulting in a predictable design choice aimed at optimization utilization of processor resources.
Regarding claim 42, claim 42 has similar limitations as of claim 41, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 41.
Regarding claim 43, claim 43 has similar limitations as of claim 41, except it is a system claim (Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer program, computer-readable recording media), therefore it is rejected under the same rationale as claim 41.
Claim(s) 6, 8, 14, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (US 20190110076), hereinafter referred to as “Lim’, in view of Khan et al. "Software architecture of high efficiency video coding for many-core systems with power-efficient workload balancing." In 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1-6. IEEE, 2014, hereinafter referred to as “Khan”, and in further view of Phillips et al. (US 20200107003), hereinafter referred to as “Phillips”.
Regarding claim 6, Lim in view of Khan discloses the apparatus of claim 5, but does not disclose wherein the tile generation circuitry is to generate the first input tiles and the second input tiles based on a field of view associated with a client device.
In the same art of stitching and encoding 360° frames, Phillips discloses wherein the tile generation circuitry is to generate the first input tiles and the second input tiles based on a field of view associated with a client device (Par. 0005 and Par. 0085; the video quality of all tiles of the tiled video frame may be equalized to a highest video quality corresponding to the quality of the tiles presented in a viewport of the client device).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the parallel tile-based architecture of Lim in view of Khan with Phillip’s client and field of view-based tile generation. The motivation lies in the advantage of enhancing performance in client devices such as VR, and prioritizing relevant visual content based on a user’s field of view. This modification yields predictable results in reducing bandwidth usage and improving rendering efficiency for an immersive user experience (Phillips Par. 0005; optimized 360° immersive video viewing experiences including, inter alia, providing client-based quality control in 360° immersive video).
Regarding claim 8, Lim in view of Khan discloses the apparatus of claim 7, but does not disclose wherein the encoding circuitry is to operate on the respective first isolated memory blocks and the respective second isolated memory blocks.
In the same art of stitching and encoding 360° frames, Phillips discloses the apparatus of claim 7 and further discloses wherein the encoding circuitry is to operate on the respective first isolated memory blocks and the respective second isolated memory blocks (Phillips Par. 0119-0120 and Fig. 13C; The data of the tiles (either from the BIE streams or from the PE streams) selected based on a tile selection process is copied in a memory (block 1364). Also see Par. 0152 and 0159 on heap memory structure) in parallel (Phillips Par. 0076 and 0084; parallel processing).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Phillip’s isolated memory block processing into the parallel tile-based architecture of Lim in view of Khan. The motivation lies in the advantage of improving parallel execution efficiency when processing independent tiles concurrently. Such architectural isolation is a common technique in high throughput multimedia system and would yield predictable results in enhanced scalability and performance.
Regarding claim 14, claim 14 has similar limitations as of claim 6, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 6.
Regarding claim 16, claim 16 has similar limitations as of claim 8, except it is a CRM claim (Lim Par. 0006-0007; processing circuitry and memory connected to the processing circuitry…computer-readable recording media), therefore it is rejected under the same rationale as claim 8.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNY NGAN TRAN whose telephone number is (571)272-6888. The examiner can normally be reached Mon-Thurs 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNY N TRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights