Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Remarks
Applicant’s arguments have been fully considered but are moot due to the new ground of rejection below to address the amended subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-8, 10-12, 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chaturvedi (US Patent 10,210,664) in view of Skidmore (US 2020/0074725)
Claim 1
Chaturvedi discloses a control method based on extended reality (Col. 1: “In the above example augmented reality UI, an augmented reality view of the image data can be provided on the screen and can include the changes to the first lighting profile.”), comprising:
obtaining a real environment image (“Sub-process 502, via the processor of the computing device, receives image data of a live camera view from the camera.”) and a spatial positional relationship of a real environment (“change in the relative position of the user with respect to the computing device.”)
rendering (“render graphical changes…”), (“The 3D lighting model provides immersive experience in augmented reality when a user adds lighting to a screen view or UI and moves the screen view or UI to see effects of the added lighting in a different (e.g., third dimension) of the 2D screen”), and
displaying the rendered extended reality space (“view or UI to see effects of the added lighting in a different (e.g., third dimension) of the 2D screen”)
simulating an illumination effect of the virtual light source on an object of the real environment in the extended reality space (Chaturvedi: “simulation of an immersive experience in three-dimension (3D) for the representation of objects via the 2D screen of an augmented realty UI…For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source….Further, the objects 304, 306 are presented with textural shading, shadowing, and/or reflections as captured by the camera. When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”).
Chaturvedi does not explicitly disclose, but Skidmore discloses:
mapping the real environment to an extended reality space based on the real environment image and the spatial positional relationship, the mapping comprising generating a two- dimensional or three-dimensional model corresponding to the real environment in an extended reality device to display the real environment in the extended reality space (Skidmore, ¶ 12: “Virtual model…corresponding virtual model may also be characterized as 3D but need not necessarily be so (i.e., a model may be two-dimensional…correspond with real world locations. In other words, the virtual model may include a virtual landscape modeled after the real world landscape.”);
and rendering, based on a light source parameter of a virtual light source in the extended reality space (Skidmore, ¶ 17: “virtual world 111 has the result that the real world parameters 121 and virtual world parameters 122 may be related, in some cases very closely related. As was the case with the real world parameters 121, the virtual world parameters 122 of interest may be capable of influencing light conditions (though all parameters determined need not necessarily be so)”), a light effect for the two-dimensional or three-dimensional model mapped to the extended reality space based on a light source parameter of the virtual light source (Skidmore, ¶ 19: “the AR system proceeds with determining lighting and shadow parameters 123 for the sourced object(s) based on the determined parameters. The resulting parameters may then be supplied to the output device 131 at the output stage 130.”), and
displaying the rendered extended reality space (Skidmore, ¶ 20: “An output device 131 (e.g., an AR device) may start with a completely (i.e., 100%) real view 132 or an existing AR view (i.e., containing both real content and virtual content). The output device 131 uses the information carried over from the process stage 120 to render one or more augmentations 134 in an AR view 133 for the end user's consumption”), comprising:
simulating an illumination effect of the virtual light source on an object of the real environment in the extended reality space based on the light source parameter of the virtual light source (Chaturvedi: “For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source….Further, the objects 304, 306 are presented with textural shading, shadowing, and/or reflections as captured by the camera. When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider mapping and light source parameters as claimed.
One of ordinary skill in the art would have motivation to “This approach advantageously leverages how virtual content is stored and uses the storage technique to improve the authenticity of augmentations sourced from such storage”(Skidmore, ¶ 7). One of ordinary skill in the art would have had a reasonable expectation of success because Chaturvedi considers lighting simulation in the same field of endeavor.
Claim 2
Chaturvedi discloses the method further comprises: obtaining morphological data of an environmental object in the real environment (“A structured lighting example includes projecting a regular pattern of light (e.g., IR light) to the objects and analyzing the changes in the reflected pattern due to the 3D features of the objects.”), initial lighting for environment (“Further, in an example, a similar method from a scene or view capture to create light effect 312, as described above, is applicable to capture or build light profiles for lighting source 308. For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source”), and/or initial lighting for environmental object (“The light information can be analyzed to determine a first lighting profile associated with the representation of the object. Position information associated with the object with respect to the camera can be determined.”).
Claim 3
Chaturvedi discloses wherein, the mapping the real environment to an extended reality space based on the spatial positional relationship comprises: mapping the real environment to the extended reality space based on one or more of the spatial positional relationship, the morphological data of the environmental object in the real environment, the initial lighting for environment, or the initial lighting for environmental object (“Further, in an example, a similar method from a scene or view capture to create light effect 312, as described above, is applicable to capture or build light profiles for lighting source 308. For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source. When the light source is added, as in FIG. 3B, the light profile associated with the light source 308, and stored in a database, is provided to overlay or to change the pixel properties of the surrounding areas of the added light source 308 and table 304, for instance.”)
Claim 4
Chaturvedi discloses, wherein, the method further comprises: calibrating an optical property of an environmental object mapped to the extended reality space (“The method and system herein, in an example, addresses the light surrounding each of these representations of the objects 124, 126, and any light interactions, such as reflections, etc.”)
Claim 5
Chaturvedi discloses wherein, the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises: rendering the light effect for the real environment mapped to the extended reality space based on the light source parameter and the optical property of the environmental object (“When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”)
Claim 6
Chaturvedi discloses wherein the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises:
rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source and reference data (“When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”), wherein the reference data comprises one or more of:
a spatial dimension of the real environment (“The 2D and 3D image data, in an example, forms the basis of ambient lighting 302”), a positional relationship of environmental objects in the real environment (“The direction is in reference to the light direction for an object with respect to the camera of the computing device 320”), a material of an environmental object in the real environment (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”), or a user posture (“As the user tilts the device, the relative position of the light source can be tracked using the same detection approaches discussed elsewhere herein”).
Claim 7
Chaturvedi discloses wherein the light source parameter comprises one or more of the followings of the virtual light source: a light source position, a light source angle, a light source intensity, a hue, a focal length, a softness, and a shading degree (“The second lighting profile provides, in an example, at least intensity values and direction information for light projected from the light source.”)
Claim 8
Chaturvedi discloses wherein, the method further comprises: setting a virtual reflective object in the extended reality space (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”)
Claim 10
Chaturvedi discloses wherein, the real environment comprises a photographic scene and a photographic object, and the virtual light source comprises a lighting source for the photographic object (Chaturvedi, Col. 3: “Accordingly, the systems and methods herein provide a richer experience in augmented reality or image/video modes on a screen, where additional representations of objects (e.g., tables and couches in photos or videos) to a captured view, image, or video is not merely a passive addition, but actively changes aspects of the representations of these objects. In an embodiment, the active changes can include changing a lighting surrounding one or more representations of objects using virtual lighting sources added to the captured view, image, or video.”)
Claim 11
Chaturvedi discloses wherein, the real environment comprises artwork (Chaturvedi, Col. 3: “FIGS. 1A-1C illustrate an example approach for providing a light source in a live camera view in accordance with an embodiment. FIG. 1A provides a view or scene 100 for capture using a camera or any image capture element (e.g., reference numeral 130 of FIG. 1B) of a computing device (e.g., reference numeral 128 of FIG. 1B). The scene includes objects, such as tables 104, 106, a couch and a picture frame.”), and the virtual light source comprises a light source of the artwork (Chaturvedi, Col. 3: “changing a lighting surrounding one or more representations of objects using virtual lighting sources added to the captured view, image, or video. This process allows a user to preview, for example, effects from a virtual light source added into a live camera view, an image, or a video on a screen.”).
Claim 12
Chaturvedi discloses wherein, the real environment comprises furniture (Chaturvedi, Col. 4: “A representation of an object may be the live camera view capture of furniture and human subjects in a field of view of the camera”), and the virtual light source comprises a lamp (Chaturvedi, Col. 4: “augmenting a light profile of a physical space represented in a live camera view with a light profile associated with a virtual lamp”).
Claim 14
The same teachings and rationales in claim 1 are applicable to claim 14, with Chatuvedi disclosing an electronic device comprising: at least one memory and at least one processor; wherein the at least one memory is configured to store program code, and the at least one processor is configured to call the program code stored in the at least one memory to execute a method comprising (Fig. 4: “In an example utilizing the network architecture 400, a computing device 402 includes a processor, a screen, a camera, and memory including instructions that the processor executes to perform processes described in this disclosure.”)
Claim 15
The same teachings and rationales in claim 2 are applicable to claim 15.
Claim 16
The same teachings and rationales in claim 3 are applicable to claim 16.
Claim 17
The same teachings and rationales in claim 4 are applicable to claim 17.
Claim 18
The same teachings and rationales in claim 5 are applicable to claim 18.
Claim 19
The same teachings and rationales in claim 6 are applicable to claim 19.
Claim 20
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Claim 20 as drafted recites a non-transitory computer-readable storage medium…
Because non-transitory explicitly excludes transitory propagating signals and the like, the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 1 are applicable to claim 20, with Chatuvedi disclosing a non-transitory computer readable medium as claimed (Col. 1: “Include computer-readable medium storing instructions that, when executed by a processor of the server, allow the server to perform its intended functions.”)
Claim(s) 1-8, 10-12, 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chaturvedi (US Patent 10,210,664) in view of Skidmore (US 2020/0074725) and Boulton (US 2018/0182160)
Claim 9
Chaturvedi as modified by Skidmore does not explicitly disclose, but Boulton discloses wherein, the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises: determining a reflection of the reflective object based on an optical property of the reflective object; and rendering the real environment image mapped to the extended reality space based on the virtual light source and the reflection of the reflective object (Boulton, ¶ 45: “In some cases, the lighting effects that should be applied to a virtual object can change over time, either because lighting conditions of the physical environment change, or one or more properties of the virtual object (e.g., shape, size, position, color, and/or reflectivity) change over time.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider reflectivity as claimed.
One of ordinary skill in the art would have motivation to coordinate the positions of the virtual and real environment for the purpose of adding virtual elements, such as lighting and accounting for real world properties of materials in the scene (“In some cases, the lighting effects that should be applied to a virtual object can change over time, either because lighting conditions of the physical environment change, or one or more properties of the virtual object (e.g., shape, size, position, color, and/or reflectivity) change over time.”))(See Boulton, ¶ 45). This is further suggested by Chaturvedi (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”). One of ordinary skill in the art would have had a reasonable expectation of success because Boulton’s techniques are applicable to lighting and Chaturvedi utilizes lighting relative to the real environment.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chaturvedi (US Patent 10,210,664) in view of Skidmore (US 2020/0074725) and Delamont (US 2020/0368616)
Claim 13
Chaturvedi does not disclose, but Delamont discloses wherein, the real environment comprises a real game scene, and the virtual light source comprises a virtual lighting prop (Delamont, ¶ 559, 579: “the game server 88 or host 89 may user external 3D projection mapping techniques together with its external projectors 96, to project lighting effects and augmented images of the IR Laser Beam, in which using same state information captured from the users IR Laser Gun Apparatus 47 these virtual images and lighting effects may be accurately augmented over real-world objects or surfaces.. a user's physical hand and arm movements this shall invoke the re-rendering of its surfaces to apply lighting and shading visual effects in real-time according to other real-game objects or virtual game objects world coordinates, directional vectors and relative directional lighting to that of the real world devices/game object world coordinates, directional vector and directional lighting.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a game scene.
One of ordinary skill in the art would have motivation to coordinate the positions of the virtual and real environment for the purpose of adding virtual elements, such as lighting to different real world objects. One of ordinary skill in the art would have had a reasonable expectation of success because Chaturvedi and Boulton’s techniques are applicable to lighting of any real objects and Delamont utilizes lighting relative to the real environment to improve the game experience. .
Additional Prior Art
Additional prior art relevant to Applicant’s disclosure but not relied upon:
Mori (US 2024/0087156) also considers lighting in the same context (“The information processing device 10 in this example may arrange a virtual light source at a position included in the virtual space and corresponding to a light source position input from the corresponding sensor device 30, and light the virtual object. In this configuration, lighting considering the position of the light source in the real space is also applied to the virtual object within the virtual space.”)(See ¶ 74)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights