DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
The Objection of Claims 16 and 17 for the limitation “spatially upscale the recorded peripheral region to be viewed by the subsequent user to a resolution higher than the second resolution” is withdrawn in light of the amendment to at least Claim 16.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 2, 6, 8 – 9, and 15 – 17 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (U.S. PG Pub 2017/0244775) in view of Shpunt (U.S. PG Pub 2018/0081178) in view of Taylor, Jr. et al. (U.S. PG Pub 2016/0292589).
Regarding Claim 1, Ha et al. teach a video processing method comprising:
obtaining a circular panoramic video (Figure 4A, Element 430. Paragraph 23) recording that comprises one or more images comprising:
a region (Figure 6A, Element 603. Paragraphs 24 and 49) corresponding to a head of a user (Figure 1, Element 200A and Figure 6A, Element 601. Paragraphs 28 and 49), the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a first resolution (Element first resolution. Paragraph 24), and
a peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) outside the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a second resolution (Element second resolution. Paragraph 24) that is lower than the first resolution (Element first resolution. Paragraph 24); and
performing spatial upscaling (Paragraph 54) of the peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) to a resolution (Element first resolution. Paragraph 24) higher than the second resolution (Element second resolution. Paragraph 24) and the second region being peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49); and
displaying the region (Figure 6A, Element 603. Paragraphs 24 and 49) and the upscaled (Paragraph 54) peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) to a subsequent user (Figure 1, Element 200B. Paragraph 28).
Ha et al. is silent with regards to an originally displayed region corresponding to a head of a user, the originally displayed region recorded at a first resolution; and a peripheral region outside the original displayed region recorded at a second resolution that is lower than the first resolution; and performing spatial upscaling to a second resolution at least by applying a plurality of machine learning models to the second region, at least some of the plurality of machine learning models being smaller or simpler than others of the plurality of machine learning models.
Shpunt teaches an originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) corresponding to a head of a user (Paragraph 29), the originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded (Figure 1, Elements 140 and 150. Paragraph 33) at a first resolution (Figure 3, Element 140. Paragraph 33); and a peripheral region (Figure 3, Element 310 outside of region 320. Paragraph 53) outside the original displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded at a second resolution (Figure 3, Element 150. Paragraph 33) that is lower than the first resolution (Figure 3, Element 140. Paragraph 33).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the virtual reality device of Ha et al. with the video capturing system of Shpunt. The motivation to modify the teachings of Ha et al. with the teachings of Shpunt is to capture image data using more than one resolution, such as to possibly reduce overall overhead, cost, and workload, as taught by Shpunt (Paragraph 6).
Taylor, Jr. et al. teach the second resolution at least by applying a plurality of machine learning models (Paragraphs 80 – 90) to the second region (Figure 9. Paragraphs 112 - 117), at least some of the plurality of machine learning models (Paragraphs 80 – 90) being smaller or simpler (Paragraph 117. Taylor, Jr. et al. discloses “In some embodiments, the vector quantization codebook 801 includes only the top layer (layer 3, in the embodiment of FIG. 8) of the machine learned model parameters. In some embodiments in which the number of levels of the machine learned model parameters are adjustable in the compression process, the vector quantization codebook includes multiple layers of the machine learned model parameters. Including only the top layer may reduce the memory and processing power required to store and run the image compression application. Including multiple layers allows for tunable levels of compression (Paragraph 117. Emphasis Added).”) than others of the plurality of machine learning models (Paragraphs 80 – 90).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al. and the video capturing system of Shpunt with the machine learning of Taylor, Jr. et al. The motivation to modify the teachings of Ha et al. and Shpunt with the teachings of Taylor, Jr. et al. is to provide model parameters to compress and decompress digital images at compression ratios unachievable by conventional methods, Taylor, Jr. et al. (Paragraph 11).
Regarding Claim 2, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. teach wherein the spatial upscaling (Paragraph 54) is to a resolution (Element first resolution. Paragraph 24) substantially equal to that of the first resolution (Element first resolution. Paragraph 24).
Regarding Claim 6, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. teach comprising:
for at least a predetermined number of preceding frames, storing a position of (Figure 8A, Element 422. Paragraph 79) at least a subset of image data having a resolution (Element first resolution. Paragraph 24) higher than the second resolution (Element second resolution. Paragraph 24) in each respective video frame; and
when performing spatial upscaling (Paragraph 54) of a given part of a current frame of the circular panoramic video (Figure 4A, Element 430. Paragraph 23), using image data of one or more preceding frames having the higher resolution (Element first resolution. Paragraph 24) at the position of the given part of the current frame as an input (Paragraph 54).
Regarding Claim 8, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. teach comprising:
generating a reference circular panoramic image (Figure 4A, Element 410. Paragraphs 23 - 24) using at least a subset of image data having a resolution higher (Element first resolution. Paragraph 24) than the second resolution (Element second resolution. Paragraph 24) in each of a predetermined number of preceding respective video frames, the circular panoramic image (Figure 4A, Element 430. Paragraph 23) thus storing the most recently rendered higher resolution (Element first resolution. Paragraph 24) rendered pixels in each direction on the reference circular panoramic image (Figure 4A, Element 410. Paragraphs 23 - 24); and
when performing spatial upscaling (Paragraph 54) of a given part of a current frame of the circular panoramic video (Figure 4A, Element 430. Paragraph 23), using image data from a corresponding part of the reference circular panoramic image (Figure 4A, Element 410. Paragraphs 23 - 24) as an input.
Regarding Claim 9, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 8 (See Above). Ha et al. teach wherein pixel data for a respectively higher resolution region (Figure 8A, Element 621. Paragraph 79) of a given image frame is stored by the reference circular panoramic image (Figure 4A, Element 410. Paragraphs 23 - 24) in preference to (Figure 8A. Paragraph 79) pixel data for a respectively lower resolution region (Figure 8A, Elements 624 - 625. Paragraph 79).
Regarding Claim 15, Ha et al. teach a non-transitory, computer readable storage medium (Paragraph 123) containing a computer program comprising computer executable instructions, which when executed by a computer system, cause the computer system to perform a video processing method comprising:
obtaining a circular panoramic video (Figure 4A, Element 430. Paragraph 23) recording that comprises one or more images comprising:
a region (Figure 6A, Element 603. Paragraphs 24 and 49) corresponding to a head of a user (Figure 1, Element 200A and Figure 6A, Element 601. Paragraphs 28 and 49), the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a first resolution (Element first resolution. Paragraph 24), and
a peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) outside the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a second resolution (Element second resolution. Paragraph 24) that is lower than the first resolution (Element first resolution. Paragraph 24); and
performing spatial upscaling (Paragraph 54) of the recorded peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) to a resolution (Element first resolution. Paragraph 24) higher than the second resolution (Element second resolution. Paragraph 24); and
displaying another circular panoramic video (Figure 4A, Element 430. Paragraph 23) comprising the region (Figure 6A, Element 603. Paragraphs 24 and 49) and the upscaled (Paragraph 54) peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) to a subsequent user (Figure 1, Element 200B. Paragraph 28).
Ha et al. is silent with regards to an originally displayed region corresponding to a head of a user, the originally displayed region recorded at a first resolution; and a peripheral region outside the original displayed region recorded at a second resolution that is lower than the first resolution; and performing spatial upscaling to a second resolution at least by applying a plurality of machine learning models to the second region, at least some of the plurality of machine learning models being smaller or simpler than others of the plurality of machine learning models.
Shpunt teaches an originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) corresponding to a head of a user (Paragraph 29), the originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded (Figure 1, Elements 140 and 150. Paragraph 33) at a first resolution (Figure 3, Element 140. Paragraph 33); and a peripheral region (Figure 3, Element 310 outside of region 320. Paragraph 53) outside the original displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded at a second resolution (Figure 3, Element 150. Paragraph 33) that is lower than the first resolution (Figure 3, Element 140. Paragraph 33).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the virtual reality device of Ha et al. with the video capturing system of Shpunt. The motivation to modify the teachings of Ha et al. with the teachings of Shpunt is to capture image data using more than one resolution, such as to possibly reduce overall overhead, cost, and workload, as taught by Shpunt (Paragraph 6).
Taylor, Jr. et al. teach the second resolution at least by applying a plurality of machine learning models (Paragraphs 80 – 90) to the second region (Figure 9. Paragraphs 112 - 117), at least some of the plurality of machine learning models (Paragraphs 80 – 90) being smaller or simpler (Paragraph 117. Taylor, Jr. et al. discloses “In some embodiments, the vector quantization codebook 801 includes only the top layer (layer 3, in the embodiment of FIG. 8) of the machine learned model parameters. In some embodiments in which the number of levels of the machine learned model parameters are adjustable in the compression process, the vector quantization codebook includes multiple layers of the machine learned model parameters. Including only the top layer may reduce the memory and processing power required to store and run the image compression application. Including multiple layers allows for tunable levels of compression (Paragraph 117. Emphasis Added).”) than others of the plurality of machine learning models (Paragraphs 80 – 90).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al. and the video capturing system of Shpunt with the machine learning of Taylor, Jr. et al. The motivation to modify the teachings of Ha et al. and Shpunt with the teachings of Taylor, Jr. et al. is to provide model parameters to compress and decompress digital images at compression ratios unachievable by conventional methods, Taylor, Jr. et al. (Paragraph 11).
Regarding Claim 16, Ha et al. teach a video processing system comprising:
one or more processors configured to:
obtaining a circular panoramic video (Figure 4A, Element 430. Paragraph 23) recording that comprises one or more images comprising:
a region (Figure 6A, Element 603. Paragraphs 24 and 49) corresponding to a head of a user (Figure 1, Element 200A and Figure 6A, Element 601. Paragraphs 28 and 49), the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a first resolution (Element first resolution. Paragraph 24), and
a peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) outside the region (Figure 6A, Element 603. Paragraphs 24 and 49) recorded at a second resolution (Element second resolution. Paragraph 24) that is lower than the first resolution (Element first resolution. Paragraph 24); and
spatially upscale (Paragraph 54) the recorded peripheral region (Figure 6A, Element 604. Paragraphs 24 and 49) to be viewed by a subsequent user (Figure 1, Element 200B. Paragraph 28) to a resolution (Element first resolution. Paragraph 24) higher than the second resolution (Element second resolution. Paragraph 24).
Ha et al. is silent with regards to an originally displayed region corresponding to a head of a user, the originally displayed region recorded at a first resolution; and a peripheral region outside the original displayed region recorded at a second resolution that is lower than the first resolution.
Shpunt teaches an originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) corresponding to a head of a user (Paragraph 29), the originally displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded (Figure 1, Elements 140 and 150. Paragraph 33) at a first resolution (Figure 3, Element 140. Paragraph 33); and a peripheral region (Figure 3, Element 310 outside of region 320. Paragraph 53) outside the original displayed (Figure 1, Element 170. Paragraph 32) region (Figure 3, Element 320. Paragraph 53) recorded at a second resolution (Figure 3, Element 150. Paragraph 33) that is lower than the first resolution (Figure 3, Element 140. Paragraph 33).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the virtual reality device of Ha et al. with the video capturing system of Shpunt. The motivation to modify the teachings of Ha et al. with the teachings of Shpunt is to capture image data using more than one resolution, such as to possibly reduce overall overhead, cost, and workload, as taught by Shpunt (Paragraph 6).
Taylor, Jr. et al. teach the second resolution at least by applying a plurality of machine learning models (Paragraphs 80 – 90) to the second region (Figure 9. Paragraphs 112 - 117), at least some of the plurality of machine learning models (Paragraphs 80 – 90) being smaller or simpler (Paragraph 117. Taylor, Jr. et al. discloses “In some embodiments, the vector quantization codebook 801 includes only the top layer (layer 3, in the embodiment of FIG. 8) of the machine learned model parameters. In some embodiments in which the number of levels of the machine learned model parameters are adjustable in the compression process, the vector quantization codebook includes multiple layers of the machine learned model parameters. Including only the top layer may reduce the memory and processing power required to store and run the image compression application. Including multiple layers allows for tunable levels of compression (Paragraph 117. Emphasis Added).”) than others of the plurality of machine learning models (Paragraphs 80 – 90).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al. and the video capturing system of Shpunt with the machine learning of Taylor, Jr. et al. The motivation to modify the teachings of Ha et al. and Shpunt with the teachings of Taylor, Jr. et al. is to provide model parameters to compress and decompress digital images at compression ratios unachievable by conventional methods, Taylor, Jr. et al. (Paragraph 11).
Regarding Claim 17, Ha et al. in view of Shpunt teach the video processing system according to claim 16 (See Above). Ha et al. teach wherein the one or more processors (Figure 2, Element 370. Paragraph 37) are configured to output the video for display to a user (Paragraph 23).
Claims 3 – 5, 7, and 10 – 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (U.S. PG Pub 2017/0244775) in view of Shpunt (U.S. PG Pub 2018/0081178) in view of Taylor, Jr. et al. (U.S. PG Pub 2016/0292589) in view of Nguyen et al. (U.S. PG Pub 2017/0236252).
Regarding Claim 3, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. is silent with regards to wherein the originally displayed region further comprises a foveal region at a third resolution higher than the first resolution, and the method comprises: performing spatial upscaling of the originally displayed region at the first resolution to substantially the third resolution.
Nguyen et al. teach wherein the originally displayed region (Figure 7D, Element 62. Paragraph 113) further comprises a foveal region (Figure 7D, Element 60. Paragraph 113) at a third resolution higher (Figure 7D, Element 8K. Paragraph 113) than the first resolution (Figure 7D, Element 4K. Paragraph 113), and the method comprises: performing spatial upscaling (Paragraph 41 and 123) of the originally displayed region (Figure 7D, Element 62. Paragraph 113) at the first resolution (Figure 7D, Element 62. Paragraph 113) to substantially the third resolution (Figure 7D, Element 8K. Paragraph 113).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Regarding Claim 4, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. is silent with regards to wherein the circular panoramic video recording comprises at least one respective transitional region provided between one or more of a foveal region and the peripheral region; and wherein the at least one respective transitional region has a resolution between the resolutions of the one or more of a foveal region and the peripheral region.
Nguyen et al. teach wherein the circular panoramic video recording comprises at least one respective transitional region (Figure 7D, Element 64. Paragraph 113) provided between one or more of a foveal region (Figure 7D, Element 60. Paragraph 113) and the peripheral region (Figure 7D, Element 66. Paragraph 113); and
the at least one respective transitional region (Figure 7D, Element 64. Paragraph 113) has a resolution between (Paragraph 113) the resolutions of the one or more of a foveal region (Figure 7D, Element 60. Paragraph 113) and the peripheral region (Figure 7D, Element 66. Paragraph 113).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Regarding Claim 5, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. is silent with regards to wherein the spatial upscaling is performed by a machine learning system trained on input image data at a lower input resolution from among the recording resolutions and corresponding target image data at a higher output resolution among the recording resolutions.
Nguyen et al. teach wherein the circular panoramic video recording (Figure 1A, Element 150. Paragraph 50) comprises at least a first respective transitional region (Figure 7D, Element 64. Paragraph 113) provided between one or more of a foveal region and original field of view region, and the original field of view region (Figure 7D, Element 62. Paragraph 113) and the further peripheral region (Figure 7D, Element 66. Paragraph 113); and the respective transitional region (Figure 7D, Element 64. Paragraph 113) having a resolution in between the resolutions of the two regions it transitions between (Paragraph 113).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Regarding Claim 7, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. is silent with regards to wherein the originally displayed region further comprises a foveal region at a third resolution higher than the first resolution, and the method further comprises: for at least a predetermined number of preceding frames, storing a position of image data of at least the third resolution in each respective video frame; and when performing spatial upscaling of a given part of a current frame of the circular panoramic video, using image data of one or more preceding frames having at least the third resolution at the position of the given part of the current frame as an input.
Nguyen et al. teach wherein the originally displayed region (Figure 7D, Element 62. Paragraph 113) further comprises a foveal region (Figure 7D, Element 60. Paragraph 113) at a third resolution (Figure 7D, Element 8K. Paragraph 113) higher than the first resolution (Figure 7D, Element 4K. Paragraph 113), and the method further comprises: for at least a predetermined number of preceding frames, storing the position of image data (Paragraphs 45 – 47) of at least the third resolution (Figure 7D, Element 8K. Paragraph 113) in each respective video frame (Figure 1A, Element 150. Paragraph 50); and when performing spatial upscaling (Paragraph 41 and 123) of a given part of a current frame of the circular panoramic video (Figure 1A, Element 150. Paragraph 50), using image data of one or more preceding frames having at least the third resolution (Figure 7D, Element 8K. Paragraph 113) at the position of the given part of the current frame as an input (Paragraphs 45 – 47).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Regarding Claim 10, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 8 (See Above). Ha et al. is silent with regards to wherein the spatial upscaling is performed by a machine learning system trained on input image data at a lower input resolution from among the recording resolutions together with corresponding input data from the reference circular panoramic image, and corresponding target image data at a higher output resolution among the recording resolutions.
Nguyen et al. teach wherein the spatial upscaling (Paragraph 41 and 123) is performed by a machine learning system (Paragraphs 46) trained on input image data at a lower input resolution (Figure 7D, Element 4K. Paragraph 113) from among the recording resolutions together with corresponding input data from the reference circular panoramic image (Figure 1A, Element 150. Paragraph 50), and corresponding target image data at a higher output resolution (Figure 7D, Element 8K. Paragraph 113) among the recording resolutions.
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al. and the video capturing system of Shpunt with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al. and Shpunt with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Regarding Claim 11, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. is silent with regards to wherein the one or more images of the circular panoramic video are rendered using a cube map, and the spatial upscaling is performed by a plurality of machine learning systems trained on one or more respective facets of the cube map.
Nguyen et al. teach wherein the one or more images of the circular panoramic video (Figure 1A, Element 150. Paragraph 50) are rendered using a cube map (Paragraphs 65 – 66), and the spatial upscaling (Paragraph 41 and 123) is performed by a plurality of machine learning systems (Paragraphs 46) trained on one or more respective facets of the cube map (Paragraphs 65 – 66).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al. and the video capturing system of Shpunt with the foveated view of Nguyen et al. The motivation to modify the teachings of Ha et al. and Shpunt with the teachings of Nguyen et al. is to provide a spherical display based on the plurality of foveated views in the viewing region, as taught by Nguyen et al. (Paragraph 6).
Claims 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (U.S. PG Pub 2017/0244775) in view of Shpunt (U.S. PG Pub 2018/0081178) in view of Taylor, Jr. et al. (U.S. PG Pub 2016/0292589) in view of Plowman (U.S. PG Pub 2014/0139542).
Regarding Claim 14, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the video processing method according to claim 1 (See Above). Ha et al. teach wherein: the circular panoramic video (Figure 4A, Element 430. Paragraph 23) recording comprises the display.
Ha et al. is silent with regards to the display comprises a record of an originally displayed region of each frame, the method comprising: during playback, displaying a visual indication of where the original field of view was within the display when the subsequent user’s own field of view diverges from the original field of view by a threshold amount.
Plowman teaches the display (Figure 1, Element 106. Paragraph 11) comprises a record of an original field of view (Element Previous gaze area. Paragraph 18) region of each frame, the method comprising: during playback, displaying a visual indication (Element Outline. Paragraph 18) of where the original field of view (Element Previous gaze area. Paragraph 18) was within the display (Figure 1, Element 106. Paragraph 11) when the subsequent user’s own field of view (Figure 1, Element 122. Paragraph 13) diverges from the original field of view (Element Previous gaze area. Paragraph 18) by a threshold amount (Element eye movement. Paragraph 18).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the visual indication of Plowman. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Plowman is to outline where the user’s attention was previously focused, as taught by Plowman (Paragraph 18).
Regarding Claim 18, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the method of claim 12 (See Above). Ha et al. is silent with regards to further comprising: displaying an indicator showing either a current direction of the originally displayed region, a respective edge of a periphery of a field of view of the subsequent viewer, or a combination thereof.
Plowman teaches further comprising: displaying an indicator (Element Outline. Paragraph 18) showing either a current direction of the originally displayed region, a respective edge of a periphery of a field of view (Figure 1, Element 122. Paragraph 13) of the subsequent viewer, or a combination thereof.
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the visual indication of Plowman. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Plowman is to outline where the user’s attention was previously focused, as taught by Plowman (Paragraph 18).
Claims 19 – 21 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (U.S. PG Pub 2017/0244775) in view of Shpunt (U.S. PG Pub 2018/0081178) in view of Taylor, Jr. et al. (U.S. PG Pub 2016/0292589) in view of Stafford et al. (U.S. PG Pub 2018/0188534).
Regarding Claim 19, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the method of claim 1 (See Above). Ha et al. is silent with regards to wherein performing spatial upscaling of the peripheral region to the resolution higher than the second resolution is based on a degree of interest of the subsequent viewer associated with an object or an event in the peripheral region.
Stafford et al. teach wherein performing spatial upscaling (Paragraph 63) of the peripheral region (Figure 6, Element not labeled, but is the area of Element 602 that does not include areas 604 and 608. Paragraph 69) to the resolution higher than the second resolution is based on a degree of interest (Element priority. Paragraphs 75 – 79) of the subsequent viewer associated with an object (Figure 6, Element 614. Paragraph 79) or an event in the peripheral region (Figure 6, Element not labeled, but is the area of Element 602 that does not include areas 604 and 608. Paragraph 69).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the rendering priority values of Stafford et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Stafford et al. is to allow for rending of more important objects, as taught by Stafford et al. (Paragraph 13).
Regarding Claim 20, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the method of claim 19 (See Above). Ha et al. is silent with regards to further comprising maintaining data indicating an expected degree of interest of the subsequent user in particular objects or environmental elements.
Stafford et al. teach further comprising maintaining data indicating an expected degree of interest (Element priority values. Paragraph 102) of the subsequent user in particular objects (Figure 6, Element 614. Paragraph 79) or environmental elements (Figure 6, Element 608. Paragraph 79).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the rendering priority values of Stafford et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Stafford et al. is to allow for rending of more important objects, as taught by Stafford et al. (Paragraph 13).
Regarding Claim 21, Ha et al. in view of Shpunt in view of Taylor, Jr. et al. teach the method of claim 19 (See Above). Ha et al. is silent with regards to further comprising: rendering an area in the peripheral region corresponding to the object or the event at the resolution higher than the second resolution; and rendering the peripheral region outside of the area at a lower resolution than the resolution of the area to maintain an overall computational budget.
Stafford et al. teach further comprising: rendering an area (Figure 6, Element 608. Paragraph 79) in the peripheral region (Figure 6, Element not labeled, but is the area of Element 602 that does not include areas 604 and 608. Paragraph 69) corresponding to the object (Figure 6, Element 614. Paragraph 79) or the event at the resolution higher than the second resolution (Paragraph 70); and rendering the peripheral region (Figure 6, Element not labeled, but is the area of Element 602 that does not include areas 604 and 608. Paragraph 69) outside of the area (Figure 6, Element 608. Paragraph 79) at a lower resolution than (Paragraph 70) the resolution of the area to maintain an overall computational budget.
It would have been obvious to a person of ordinary skill in the art to modify the teachings of the spherical video of Ha et al., the video capturing system of Shpunt, and the machine learning of Taylor, Jr. et al. with the rendering priority values of Stafford et al. The motivation to modify the teachings of Ha et al., Shpunt, and Taylor, Jr. et al. with the teachings of Stafford et al. is to allow for rending of more important objects, as taught by Stafford et al. (Paragraph 13).
Allowable Subject Matter
Claim 22 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art of record fails to teach at least “wherein: the plurality of machine learning models comprise a plurality of machine learning models each corresponding to one or more of the six facets of a cube map; and a machine learning model corresponding to a behind facet of the cube map is smaller or simpler than a machine learning model corresponding to a front facet of the cube map” in combination with the other limitations of at least Claim 22 and Claim 1, from which Claim 22 depends.
Response to Arguments
All arguments are considered moot in light of the new grounds of rejection above necessitated by the applicant’s amendment to the claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Chen et al. (U.S. PG Pub 2019/0026864) disclose a super-resolution foveated rendering based on a proximity to a user focus area.
Miyaki (U.S. PG Pub 2019/0282892) discloses a set of concentric rings that vary with the amount of rendering based on proximity to the virtual character.
Ha et al. (U.S. PG Pub 2020/0184931) discloses a display apparatus that is capable of upscaling in different resolutions using machine learning and/or deep learning, similar to the instant invention
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW B SCHNIREL whose telephone number is (571)270-7690. The examiner can normally be reached Monday - Friday, 10 - 6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Boddie can be reached at 571-272-0666. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.B.S/Examiner, Art Unit 2625
/WILLIAM BODDIE/Supervisory Patent Examiner, Art Unit 2625
Prosecution Insights