Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Double Patenting
Claims 1, 10, 11, 27 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12125177. Although the claims at issue are not identical, they are not patentably distinct from each other because 18/804259 is broader but for “shielding from light regardless of orientation of the image sensor”, where a camera can be moved in 3d space and changing the orientation of the camera changes the orientation of the image sensor without affecting the light-shielding region. See mapping below.
Claims 14, 15, 16, 17, 18, 20, 21 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12125177. See the explanation for claim 1 above. 12125177 claims 2, 3, 4, 1, 1, 5, 6 are equivalent to 18/804259 claims 14, 15, 16, 17, 18, 20, 21.
18/804259
12125177
An information processing apparatus comprising: a processor; and a memory storing a program which, when executed by the processor, causes the information processing apparatus to
An information processing apparatus comprising a processor; and a memory storing a program which, when executed by the processor, causes the information processing apparatus to:
execute generating processing to generate an image of a virtual space that is superimposed on an image of a real space captured by an image sensor;
execute generating processing to generate an image of a virtual space that is superimposed on an image of a real space captured by an image sensor;
and execute adding processing to add noise to the image of the virtual space,
execute determining processing to determine a correspondence between a pixel value of the image of the virtual space and a noise amount to be added to the pixel value
and in the adding processing noise is added to the image of the virtual space, based on an image acquired in the light-shielding region.
execute adding processing to add noise to the image of the virtual space in accordance with the correspondence,
wherein the image sensor includes a region for capturing an image of the real space and a light-shielding region
wherein the image sensor includes a region for capturing an image of the real space and a light-shielding region,
shielding from light regardless of orientation of the image sensor,
a plurality of correspondences between the pixel value and the noise amount are determined in advance
in the determining processing, any one of the plurality of correspondences is selected based on an image acquired in the light-shielding region.
Allowable Subject Matter
The claim is rejected under Double Patenting above. Upon filing a TD, the claim 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The claim recites “noise is added to the image of the virtual space, based on the image acquired in the light-shielding region and temperature of the image sensor”. The claim recites that both a light-shielding region and sensor temperature are used to generate noise. The prior art does not teach these limitations in combination with the other limitations
The claim is rejected under Double Patenting above. Upon filing a TD, the Claim 18 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The claim recites “wherein a plurality of correspondences between the pixel value and the noise amount are determined in advance, and in the determining processing, any one of the plurality of correspondences is selected based on the image acquired in the light-shielding region”. The claim recites that there are a plurality of correspondences determined in advance, which are then selected from. The prior art does not teach these limitations in combination with the other limitations
The claim is rejected under Double Patenting above. Upon filing a TD, the Claim 20 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The claim recites “where magnifying processing is performed on the image of the real space, in the adding processing, noise is increased at a magnification the same as that used for the magnifying processing and the noise is added to the image”. The claim recites magnification is applied to the noise. The prior art does not teach these limitations in combination with the other limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 13-14, 17, 19, 21-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kurz U.S. Patent/PG Publication 20200273212 in view of Mizutani U.S. Patent/PG Publication 20110292210.
Regarding claim 13:
(New) An information processing apparatus comprising: a processor (Kurz [0009] In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that are computer-executable to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein.).
and a memory storing a program which, when executed by the processor, causes the information processing apparatus to execute generating processing to generate an image of a virtual space that is superimposed on an image of a real space captured by an image sensor (Kurz [0025] According to some implementations, the HMD 120 presents a computer-generated reality (CGR) experience to the user while the user is present within the physical environment 105. A computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In CGR, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that comports with at least one law of physics. For example, a CGR system may detect a person's head turning and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment).
and execute adding processing to add noise to the image of the virtual space, wherein the image sensor includes a region for capturing an image of the real space (Kurz [0039] The noise match unit 246 uses the parameterized noise model to generate noise, e.g., a noise pattern, that is similar to the noise of a particular image. In some implementations, signal-to-noise ratio (SNR) levels or other camera parameters are associated with parameters for the parameterized noise model during a learning phase and the noise match unit 246 identifies a SNR level for an image, looks up the corresponding parameters, and generates a noise pattern using the parameters.)
space, based on an image acquired in the (Kurz [0040] The CGR rendering unit 248 combines image content with virtual content to provide an CGR experience, e.g., providing a view of an CGR environment for a user. The CGR rendering unit 248 may adjust the appearance of the image content or the virtual content. In some implementations, the CGR rendering unit adds noise to the virtual content that is similar to noise of the image content with which the virtual content is being combined. The added noise may be based on a noise pattern provided by the noise match unit 246 for the corresponding image.).
Kurz does not teach a light-shielding region. In a related field of endeavor, Mizutani teaches:
and a light-shielding region shielding from light regardless of orientation of the image sensor; light-shielding region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.) since the camera can be moved in 3d space changing the orientation of the camera and image sensor without affecting the functionality light-shielding region.
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 14:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein in the adding processing, noise is added to the image of the virtual space, based on the image acquired in the (Kurz [0065] At block 608, the method 600 renders computer-generated reality (CGR) content with the virtual content with added noise. The virtual content may include noise that was added based on the noise pattern. In some implementations, the noise is added to the virtual content by tiling the noise pattern over the virtual content. In some implementations, the noise pattern is randomly tiled over the virtual content. In some implementations, the noise pattern is first weighted according to a luminance response function, e.g., one that makes use of the per pixel luminance of the rendered virtual content, and then combined with the virtual content. In some implementations, noise is only added to one or a few of the virtual objects and is not added to one or more other of the virtual objects. For example, noise may not be added to user interface elements that are not supposed to look as though the elements are imaged through the camera.)
and on image processing that is performed on the image of the real space (Kurz [0039] The noise match unit 246 uses the parameterized noise model to generate noise, e.g., a noise pattern, that is similar to the noise of a particular image. In some implementations, signal-to-noise ratio (SNR) levels or other camera parameters are associated with parameters for the parameterized noise model during a learning phase and the noise match unit 246 identifies a SNR level for an image, looks up the corresponding parameters, and generates a noise pattern using the parameters.).
Kurz does not teach a light-shielding region. In a related field of endeavor, Mizutani teaches:
image acquired in the light-shielding region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 17:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein the program, when executed by the processor, further causes the information processing apparatus to execute determining processing to determine a correspondence between a pixel value of the image of the virtual space and a noise amount to be added to the pixel value, based on the image acquired in the (Kurz [0058] FIG. 8 is a block diagram illustrating an exemplary technique for creating a parameterized noise model based on noise data. In this example, the noise model parameters are determined by determining a per pixel intensity deviation 804 using the noise data 802. The patterns of the per pixel intensity deviation can be shifted against each other and used to compute a correlation, e.g., spatial correlation 806, which provides information about the noise, e.g., measuring how blobby versus grainy the noise is in the image. The spatial correlation function provides statistical information about the noise, e.g., based on how much correlation there is in the noise when it is spatially shifted against itself. The method may perform a calculation (e.g., an optimization) to identify parameters (e.g., per pixel noise 808 and smoothing kernel 810) for a parameterized noise model (e.g., a Gaussian mixture model, a polynomial representation, parameterized histogram, etc.) to produce noise output with characteristics similar to the spatial correlation 806 or the intensity deviation distribution.).
Kurz does not teach a light-shielding region. In a related field of endeavor, Mizutani teaches:
light-shielding region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 19:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein the program, when executed by the processor, further causes the information processing apparatus to execute calculating processing to calculate a noise amount to be added to the image of the virtual space using a noise amount determined from the image acquired in the (Kurz [0048] The noise match unit 346 uses the parameterize noise model to generate noise, e.g., a noise pattern, that is similar to the noise of a particular image. In some implementations, signal-to-noise ratio (SNR) levels are associated with parameters for the parameterized noise model during a learning phases and the noise match unit 346 identifies a SNR level for an image, looks up the corresponding parameters, and generates a noise pattern using the parameters.)(Kurz [0058] FIG. 8 is a block diagram illustrating an exemplary technique for creating a parameterized noise model based on noise data. In this example, the noise model parameters are determined by determining a per pixel intensity deviation 804 using the noise data 802. The patterns of the per pixel intensity deviation can be shifted against each other and used to compute a correlation, e.g., spatial correlation 806, which provides information about the noise, e.g., measuring how blobby versus grainy the noise is in the image. The spatial correlation function provides statistical information about the noise, e.g., based on how much correlation there is in the noise when it is spatially shifted against itself. The method may perform a calculation (e.g., an optimization) to identify parameters (e.g., per pixel noise 808 and smoothing kernel 810) for a parameterized noise model (e.g., a Gaussian mixture model, a polynomial representation, parameterized histogram, etc.) to produce noise output with characteristics similar to the spatial correlation 806 or the intensity deviation distribution.).
Kurz does not teach a light-shielding region. In a related field of endeavor, Mizutani teaches:
light-shielding region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 21:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein in a case where a noise amount determined from the image acquired in the light-shielding region is smaller than a threshold, noise is not added to the image of the virtual space in the adding processing, and in a case where the noise amount determined from the image acquired in the light-shielding region is larger than the threshold, noise is added to the image of the virtual space in the adding processing (Kurz [0068] Some implementations selectively add noise to virtual objects for some images but not add noise to virtual object for other images based on selection criteria. For example, the effect of/presence of noise may be more noticeable in low light conditions. Some implementations thus selectively determine to add noise to virtual objects based on determining that the light conditions are less than a particular threshold, e.g., adaptively turning on and off the virtual noise generation features. This can avoid wasting resources in bright environments in which the addition of noise to virtual objects is less noticeable.).
Regarding claim 22:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein in the adding processing, the noise is added to the image of the virtual space, based on variation of pixel values among pixels of a predetermined region, which is set to at least a part of the (Kurz [0058] FIG. 8 is a block diagram illustrating an exemplary technique for creating a parameterized noise model based on noise data. In this example, the noise model parameters are determined by determining a per pixel intensity deviation 804 using the noise data 802. The patterns of the per pixel intensity deviation can be shifted against each other and used to compute a correlation, e.g., spatial correlation 806, which provides information about the noise, e.g., measuring how blobby versus grainy the noise is in the image. The spatial correlation function provides statistical information about the noise, e.g., based on how much correlation there is in the noise when it is spatially shifted against itself. The method may perform a calculation (e.g., an optimization) to identify parameters (e.g., per pixel noise 808 and smoothing kernel 810) for a parameterized noise model (e.g., a Gaussian mixture model, a polynomial representation, parameterized histogram, etc.) to produce noise output with characteristics similar to the spatial correlation 806 or the intensity deviation distribution.).
Kurz does not teach a light-shielding region. In a related field of endeavor, Mizutani teaches:
light-shielding region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 23:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz further teaches wherein in the adding processing, the noise is added to the image of the virtual space, based on a captured image, that is acquired by the image sensor, before image processing, (Kurz [0054] At block 602, the method 600 identifies noise data (e.g., measuring camera noise in an image) for a first image captured by an image capture device.) the program, when executed by the processor, further causes the information processing apparatus to execute combination processing to generate a combined image by combining the image of the virtual space, to which noise is added by the adding processing, with the captured image and execute the image processing on the combined image (Kurz [0065] At block 608, the method 600 renders computer-generated reality (CGR) content with the virtual content with added noise. The virtual content may include noise that was added based on the noise pattern. In some implementations, the noise is added to the virtual content by tiling the noise pattern over the virtual content. In some implementations, the noise pattern is randomly tiled over the virtual content. In some implementations, the noise pattern is first weighted according to a luminance response function, e.g., one that makes use of the per pixel luminance of the rendered virtual content, and then combined with the virtual content. In some implementations, noise is only added to one or a few of the virtual objects and is not added to one or more other of the virtual objects. For example, noise may not be added to user interface elements that are not supposed to look as though the elements are imaged through the camera.).
Regarding claim 24:
(New) The information processing apparatus according to claim 13, has all of its limitations taught by Kurz in view of Mizutani. Kurz does not teach optical black. In a related field of endeavor, Mizutani teaches:
wherein the light-shielding region is an optical black region (Mizutani [0007] For a technology to measure a noise intensity of a camera, there is a method that uses, for example, an optical black. The method measures a noise intensity by providing a light-shielded area outside of an effective pixel area of a camera, measuring a luminance value of the area, and comparing the luminance value with an ordinary black level. Japanese Laid-Open Patent Publication No. 2009-296147 discusses a technology that measures a noise intensity in a camera by using the above-described optical black method and determines, by a recognition device that is provided externally of the camera, whether image recognition processing of a pick-up image is executed according to the noise intensity.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use an optical black light-shielding region as taught by Mizutani. The rationale for doing so would have been that it a simple substitution of one known element for another to obtain predictable results since Kurz has a camera that obtains noise information, and Mizutani discloses a known method of obtaining noise information using a light-shielded area, where the end result is obtaining noise information for both. Therefore it would have been obvious to combine Mizutani with Kurz to obtain the invention.
Regarding claim 25:
The claim is a/an parallel version of claim 13. As such it is rejected under the same teachings.
Regarding claim 26:
The claim is a/an parallel version of claim 13. As such it is rejected under the same teachings.
Regarding claim 27:
(New) A head mounted display comprising: (Kurz [0020] While FIGS. 1-3 depict exemplary implementations involving a head mounted device (HMD), other implementations do not necessarily involve an HMD and may involve other types of devices including, but not limited to, watches and other wearable electronic devices, mobile devices, laptops, desktops, gaming devices, home automation devices, and other devices that include or use image capture devices.).
the information processing apparatus according to claim 13 (The limitation is a/an parallel version of claim 13. As such it is rejected under the same teachings.).
and a display configured to superimpose an image of a virtual space, to which noise is added, on an image of a real space, and to display (Kurz [0065] At block 608, the method 600 renders computer-generated reality (CGR) content with the virtual content with added noise.)(Kurz [0051] FIG. 4 illustrates CGR content 400 in which the virtual content 404 (e.g., the virtual cooking appliance) does not have noise that matches the noise of the depictions of the real world content 402 (e.g., the real world cooking appliance). For example, the depiction of the real world content 402 has a noisy/grainy appearance consistent with the noise in the rest of the image. In contrast, the virtual content 404, which is a virtual version of the same type of appliance as real object 402, has a significantly smoother and otherwise less noisy appearance.).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kurz U.S. Patent/PG Publication 20200273212 in view of Mizutani U.S. Patent/PG Publication 20110292210 and Totty U.S. Patent/PG Publication 20200302681.
Regarding claim 15:
(New) The information processing apparatus according to claim 14, has all of its limitations taught by Kurz in view of Mizutani. Kurz in view of Mizutani does not teach noise reduction. In a related field of endeavor, Totty teaches:
wherein the image processing is at least one of noise reduction, edge enhancement, and contrast adjustment (Totty [0171] However, the VRVR can be otherwise modified. Examples of modifications can include: generation or modification of realistic shadows, generation or modification of realistic lighting, highlights modification or enhancements, color modifications or enhancements, noise modification or enhancements, including algorithms for adding noise, de-noising, and re-noising. ).
Therefore, it would have been obvious before the effective filing date of the claimed invention to perform noise reduction as taught by Totty. The motivation for doing so would have been to reduce the amount of noise that needs to be added, and provide a clearer picture to the user. Therefore it would have been obvious to combine Totty with Kurz in view of Mizutani to obtain the invention.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617