DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-15 are pending.
Priority
This application claims foreign priority from Japanese application JP2023-023632 filed 02/17/23.
Information Disclosure Statement
The IDS filed 02/09/24 is considered.
Response to Arguments
The Response to Election/Restriction Requirement filed 01/02/26 indicates an election without traverse. Claims 1-6 and 8-15 are pending, of which claim 7 is withdrawn.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 11, and 14-15 rejected under 35 U.S.C. 102(a)(2) as being anticipated by Guo et al. (US 20230186590 A Hereinafter “Guo”).
Regarding claim 1, Guo teaches an image processing apparatus, comprising:
one or more memories (Fig. 2, [0024]: “The end-user device 100 comprises a processor 110, a memory 120, and a 360-camera sensor 130”); and
one or more processors (Fig. 2, [0024]: “The end-user device 100 comprises a processor 110, a memory 120, and a 360-camera sensor 130”), wherein the one or more processors and the one or more memories are configured to:
input a non-perspective projection image (Fig. 4-5, [0038]: “In summary, the respective convolution layer receives omnidirectional input feature data 410 (e.g., the original omnidirectional image or features from a previous layer) which are in the omnidirectional image domain”. The omnidirectional image is the non-perspective projection image);
transform a respective image in each image region of a plurality of image regions in the non-perspective projection image into a respective perspective projection image (Fig. 5, [0038]: “The processor 110 transforms the omnidirectional input feature data 410 a plurality of input perspective feature patches 420 (e.g., squares), which are in the perspective projection image domain, using a patch-wise Equirectangular to Perspective (E2P) transform”. The feature patches act as the image regions in a plurality of image regions which are transformed from an equirectangular image to the perspective projection image); and
for each of the respective perspective projection images, after applying image processing to the respective perspective projection image (Fig. 5, [0038]: “Next, the processor 110 performs a regular convolution operation 440 on the plurality of input perspective feature patches 420 with rectified linear unit (ReLU) activation to generate a plurality of output perspective feature patches 450, which are also in the perspective projection image domain”. The regular convolution operation is a form of image processing), transform the result of the image processing into a respective non-perspective projection image (Fig. 5, [0038]: “Finally, the processor 110 transforms the plurality of output perspective feature patches 450 into omnidirectional output feature data 460, which are in the omnidirectional image domain, using a patch-wise Perspective to Equirectangular (P2E) transform 470”. The result of the image processing is transformed for the perspective projection image to the equirectangular image).
Regarding claim 11, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to:
generate a combined image based on the respective non-perspective projection images for the plurality of image regions, and output the combined image (Fig. 5, [0038]: “Finally, the processor 110 transforms the plurality of output perspective feature patches 450 into omnidirectional output feature data 460, which are in the omnidirectional image domain, using a patch-wise Perspective to Equirectangular (P2E) transform 470”. The images (patches) are processed and combined to generate the final image).
Regarding claim 14, the content of claim 14 is similar to the content of claim 1, therefore it is rejected for the same reasons of anticipation as claim 1.
Regarding claim 15, the content of claim 15 is similar to the content of claim 1, with the additional teachings of a non-transitory computer-readable storage medium. Guo also discloses this information ([0062]: “Embodiments within the scope of the disclosure may also include non-transitory computer-readable storage media or machine-readable medium for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored thereon”). Therefore, claim 15 is rejected for the same reasons of anticipation as claim 1, along with the additional teachings above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of Reed (“VIRTUAL REALITY” Hereinafter “Reed”).
Regarding claim 2, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to transform the respective image in each image region of the plurality of image regions into the respective perspective projection image (Fig. 5, [0038]: “The processor 110 transforms the omnidirectional input feature data 410 a plurality of input perspective feature patches 420 (e.g., squares), which are in the perspective projection image domain, using a patch-wise Equirectangular to Perspective (E2P) transform”. The feature patches act as the image regions in a plurality of image regions which are transformed from an equirectangular image to the perspective projection image)
While it is presumed Guo teaches the transformation of the non-perspective image based on the parameter corresponding to pixel position and the horizontal and vertical angle due to those attributes being needed to generate an equirectangular image of an omni-directional image, Guo does not expressly disclose transformation based on a parameter corresponding to a pixel position of a pixel in the image region and an angle of a vertical direction and a horizontal direction of the image region.
However, Reed teaches that in generation of equirectangular image from an omnidirectional image, parameter corresponding to a pixel position of a pixel in the image region and an angle of a vertical direction and a horizontal direction of the image region is required (Pages 3-9: Pages 3-9 describe the process for equirectangular projection, which is generating an equirectangular image from and Omnidirectional image (spherical). The image of the sphere on page 4 shows the both a parameter related to the position of the pixel is needed and the horizontal and vertical angles are needed to generate an equirectangular representation of the omnidirectional image).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Guo’s transformation process to include Reed’s process for generating the equirectangular image because such a modification is based on the use of known techniques to improve similar devices in the same way. More specifically, Reed’s equirectangular image is comparable to Guo’s equirectangular image because both are equirectangular images of omnidirectional images. Guo is silent about the process for generating an equirectangular image from an omnidirectional image. Reed provides a way to perform that generation. Therefore, it would be obvious to one of ordinary skill in the art to use the process for generating an equirectangular image in Guo’s transformation process to obtain the equirectangular image, as taught by Reed. Additionally, by using Reed’s method to obtain Guo’s equirectangular image, the transformation from the equirectangular image to a perspective image is based on the parameter associated with the pixel position and the horizontal and vertical angle of the image patches, since the generation of the equirectangular image that is transformed is based on those parameters and angles, and patches are generated in Guo, so those parameters and angles would be present in those patches. Hence, the combination of Guo and Reed teaches the claimed limitation.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of Hwang et al. (US 20190050659 A1 Hereinafter “Hwang”).
Regarding claim 3, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to
Guo does not expressly disclose applying a bilateral filter to the images.
However, Hwang teaches applying a bilateral filter to the images ([0127]: “For example, the image quality improver 230 may reduce a noise of the biometric image by applying a bilateral filter or a median filter”).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing to include Hwang’s use of a bilateral filter for image processing because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include Hwang is implicitly provided by Hwang, stating that applying the bilateral filter reduces noise in the image ([0127]: “The image quality improver 230 may apply a noise reducing algorithm as a first improvement algorithm E1 to a biometric image. For example, the image quality improver 230 may reduce a noise of the biometric image by applying a bilateral filter or a median filter”). Reduction of noise in an image improves the overall image quality. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing to include Hwang’s use of a bilateral filter for image processing with the motivation of reducing noise in the image. The person of ordinary skill in the art would have recognized the benefit of improved image quality.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of ZHU et al. (US 20230169637 A1 Hereinafter “ZHU”).
Regarding claim 4, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to
Guo does not expressly disclose applying a band-pass filter to the images.
However, ZHU teaches applying a band-pass filter to the images ([0041]: “In step 108, a bandpass filter is applied to the image from the smoothing step 106. Bandpass filtering removes high spatial frequency and low spatial frequencies, and suppresses horizontal and/or vertical stripes that may have been created by scanning of the original image (e.g., line-by-line scanning with a scanner)”).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing to include ZHU’s use of a band-pass filter for image processing because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include ZHU is implicitly provided by ZHU, stating that applying the band-pass filter removing high and low spatial frequencies ([0041]: “In step 108, a bandpass filter is applied to the image from the smoothing step 106. Bandpass filtering removes high spatial frequency and low spatial frequencies, and suppresses horizontal and/or vertical stripes that may have been created by scanning of the original image (e.g., line-by-line scanning with a scanner)”). Removal of these frequencies improves the contrast of the image, which improves the overall quality of the image. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing to include ZHU’s use of a band-pass filter for image processing with the motivation of improving contrast in the image. The person of ordinary skill in the art would have recognized the benefit of improved image quality.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of SOTODATE et al. (US 20200219234 A1 Hereinafter “SOTODATE”).
Regarding claim 5, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to
Guo does not expressly disclose perform sharpening processing, for which deep learning is used, to the images.
However, SOTODATE teaches perform sharpening processing, for which deep learning is used, to images ([0038]: “The correction and estimation unit 15 uses the image correction model to generate a sharpened image (an example of a corrected image) in which the image quality of the entire object is improved (sharpened) in consideration of the occurrence of low illuminance noise in the object during outdoor imaging (in other words, generates a high-quality image obtained by estimating and correcting a degraded portion where low illuminance noise is likely to occur)”. This correction and estimation unit uses deep learning, “The image correction model is configured by a neural network (hereinafter, referred to as “correction and estimation network”) that can realize machine learning such as deep learning” [0035]).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing to include SOTODATE’s use of deep learning sharpening for image processing because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include SOTODATE is expressly provided by SOTODATE, stating that sharpening the image improves the quality of the image ([0038]: “The correction and estimation unit 15 uses the image correction model to generate a sharpened image (an example of a corrected image) in which the image quality of the entire object is improved (sharpened) in consideration of the occurrence of low illuminance noise in the object during outdoor imaging (in other words, generates a high-quality image obtained by estimating and correcting a degraded portion where low illuminance noise is likely to occur)”. Sharpening of the image improves the quality of the image. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing to include SOTODATE’s use of deep learning sharpening for image processing with the motivation of improving sharpness in the image. The person of ordinary skill in the art would have recognized the benefit of improved image quality.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of Takeguchi et al. (US 20080192998 A1 Hereinafter “Takeguchi”).
Regarding claim 6, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to
Guo does not expressly disclose perform a convolution integration by a Gaussian filter kernel on the images.
However, Takeguchi teaches perform a convolution integration by a Gaussian filter kernel on the images ([0034]: “Where, I(x,y) is an input sectional image, G(.sigma.) is a two-dimensional Gaussian filter, L is a two-dimensional Laplacian filter, * is a symbol showing a convolution integration, F(x,y) an output image, and C is a parameter representing an amount of blur of the Gaussian filter”. The input image is processed by performing convolution integration by a gaussian filter kernel).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing to include Takeguchi’s convolutional integration using a gaussian filter kernel for image processing because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include Takeguchi is implicitly provided by Takeguchi, stating that the described formula describes the spatial filtering taking place ([0037]: “Then, the output image processed by applying the spatial filtering to the input image by the Laplacian-Of-Gaussian filter using a predetermined scale parameter .sigma. is obtained (see step A3). When a plurality of scale parameters .sigma. are set, a plurality of output images are obtained by each scale parameter respectively”). Spatially filtering the image improves the quality of the image. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing to include Takeguchi’s convolutional integration using a gaussian filter kernel for image processing with the motivation of spatially filtering the image. The person of ordinary skill in the art would have recognized the benefit of improved image quality.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of OKAMOTO et al. (US 20160093027 A1 Hereinafter “OKAMOTO”).
Regarding claim 8, Guo teaches the image processing apparatus according to claim 1,
wherein the one or more processors and the one or more memories are further configured to (Fig. 5, [0038]: “Finally, the processor 110 transforms the plurality of output perspective feature patches 450 into omnidirectional output feature data 460, which are in the omnidirectional image domain, using a patch-wise Perspective to Equirectangular (P2E) transform 470”. The result of the image processing is transformed for the perspective projection image to the equirectangular image).
Guo does not expressly disclose performing filter processing on the non-perspective
projection image that was transformed to the perspective projection image.
However, OKAMOTO teaches performing filter processing on an image using adaptive filtering ([0020]: “This achieves adaptive change of the filter strength of the lowpass filter for each pixel, depending on the edge strength of each pixel. For example, the filter strength for pixels belonging to a flat area of the image is set relatively high, which removes noise efficiently, while the filter strength of pixels belonging to an edge area of the image is set relatively low, which helps avoid smoothing of edges”. If used in Guo’s image processing step, the filter strength would be according to with a filter strength corresponding to the respective perspective projection image. That area in the respective perspective projection image is connected to a region in the non-perspective projection image, that is, there is an area in the non-perspective region that was transformed to obtain the area in the perspective projection image. So since the filter strength corresponds to the area in the perspective projection image, it corresponds to that same area in the non-perspective projection image., Hence, the combination of Guo and OKAMOTO teaches the claimed limitation).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing to include OKAMOTO’s adaptive image filtering for image processing because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include OKAMOTO is expressly provided by OKAMOTO, stating that aby applying an adaptive filter like this it can filter the image to improve quality while avoiding over smoothing of edges ([0020]: “For example, the filter strength for pixels belonging to a flat area of the image is set relatively high, which removes noise efficiently, while the filter strength of pixels belonging to an edge area of the image is set relatively low, which helps avoid smoothing of edges”). Reduction of noise in an image improves the overall image quality. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing to include OKAMOTO’s adaptive image filtering for image processing with the motivation of reducing noise in the image. The person of ordinary skill in the art would have recognized the benefit of improved image quality.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of OGURA et al. (US 20240411492 A1 Hereinafter “OGURA”).
Regarding claim 12, Guo teaches the image processing apparatus according to claim 11,
wherein the one or more processors and the one or more memories are further configured to output the combined image as an image(Fig. 5, [0038]: “Finally, the processor 110 transforms the plurality of output perspective feature patches 450 into omnidirectional output feature data 460, which are in the omnidirectional image domain, using a patch-wise Perspective to Equirectangular (P2E) transform 470”. The images (patches) are processed and combined to generate the final output image).
Guo does not expressly disclose output the image as an image for display on a head-mounted display.
However, OGURA teaches output the image as an image for display on a head-mounted display ([0002]: “Conventionally, a virtual reality (VR) technology has been proposed in which an input image such as an omnidirectional developed image which is an omnidirectional image of 360 degrees is viewed on a head mounted display or the like”. They use the word conventionally because it is conventional to display omnidirectional image on VR device/head mounted devices).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing system to include OGURA’s displaying of the process omnidirectional image using a head mounted display because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, Guo’s image processing system as modified by OGURA’s displaying of the process omnidirectional image using a head mounted display can yield a predictable result of displaying the processed omnidirectional image on a Head mounted display since OGURA teaches that displaying an omnidirectional image on a head mounted display is a conventional technique in the art. Thus, a person of ordinary skill would have appreciated including in Guo’s image processing system to include OGURA’s displaying of the process omnidirectional image using a head mounted display since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have found it obvious to use the combination of elements due to OGURA’s teachings.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20230186590 A Hereinafter “Guo”) in view of Takagi et al. (US 20080030583 A1 Hereinafter “Takagi”).
Regarding claim 13, Guo teaches the image processing apparatus according to claim 1,
Guo does not expressly disclose wherein the image processing is selected by an instruction by a user from among a plurality of image processes defined in advance.
However, Takagi teaches wherein the image processing is selected by an instruction by a user from among a plurality of image processes defined in advance (Fig. 2, [0038]: “And when, in this state, the user operates the operation unit 17 and issues a command for image processing to be performed, an image processing type selection menu as seen in the FIG. is displayed as superimposed over this image of the photographic subject. In this image processing type selection menu, a list of the above described types of image processing is displayed, and the user is able to select any one of these types of image processing by actuation of the operation unit 17”).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify Guo’s image processing system to include Takagi’s ability to allow the user to select the image processing from a list to be performed on the images because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Guo to include Takagi is implicitly provided by Takagi, stating that the user gets to select the processing type (Fig. 2, [0038]: “And when, in this state, the user operates the operation unit 17 and issues a command for image processing to be performed, an image processing type selection menu as seen in the FIG. is displayed as superimposed over this image of the photographic subject. In this image processing type selection menu, a list of the above described types of image processing is displayed, and the user is able to select any one of these types of image processing by actuation of the operation unit 17”). Allowing the user to select the image processing they want improves the user functionality alongside user quality of life. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Guo’s image processing system to include Takagi’s ability to allow the user to select the image processing from a list to be performed on the images with the motivation of improving user functionality. The person of ordinary skill in the art would have recognized the benefit of improved user functionality.
Allowable Subject Matter
Claims 9-10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
ZAKHARCHENKO et al. (US 20200118241 A1) teaches projection of an omnidirectional image into a rectangular image
Hattori (US 12462332 B2) teaches processing of fish eye images
Stalling et al. (US 7876944 B2) teaches a process of projecting an image, correcting the image, and back projecting the image.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights