Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Remarks
Applicant's arguments filed 1/16/26 have been fully considered as follows:
Applicant argues:
As asserted, Chaturvedi does not disclose or suggest feature "mapping the real environment to an extended reality space based on the real environment image and the spatial positional relationship" and thus also fails to disclose or suggest its related feature, i.e., the above distinguishing feature as now recited in amended Claim 1.
Further, Boulton discloses, e.g., paragraph [0030], method 200 includes recognizing a three-dimensional world-space position for a virtual object in the physical environment. Recognizing a three-dimensional world-space position of a virtual object includes identifying the three-dimensional world-space position in the physical environment at which the virtual object should appear, from the perspective of the wearer of the augmented reality computing device. This position may be specified in any suitable manner. In some implementations, a world-space-coordinate system may be mapped to a virtual-space-coordinate system SO that the virtual position of the virtual object in the virtual space may be mapped to a corresponding world space position in the real world.
That is, Boulton merely teaches the position mapping of virtual objects through coordinate systems. However, Boulton does not involve or suggest generation of a two-dimensional or three-dimensional model corresponding to the real environment in the device, nor does it involve displaying the real environment in the extended reality space.
(Remarks, Page 9).
Applicant’s argument is unpersuasive because applicant does not explicitly define the scope of 2D and 3D model. The plain meaning of model would include any 2D or 3D representation suitable for mapping.
Boulton in the same context considers “recognizing a three-dimensional representation may include generating the three-dimensional representation based on sensor data collected by one or more sensors” (¶ 21). Using this representation a lighting model can be created (Boulton, ¶ 32: “From three-dimensional representation 400, the augmented reality computing device generates a cube map 402 defining lighting conditions of the physical environment at the three-dimensional world-space position of the virtual object (i.e., lighting conditions of physical environment 104 at the location of virtual object 110). Because the augmented reality computing device already has the three-dimensional representation of the physical environment, the process of generating cube map 402 can be performed”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-12, 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chaturvedi (US Patent 10,210,664) in view of Boulton (US 2018/0182160)
Claim 1
Examiner’s Interpretation:
Interpretation of “two-dimensional or three-dimensional model”:
Applicant’s specification does not explicitly define the scope of the claimed models. The broadest reasonable interpretation would be any 2D or 3D representation that would allow mapping as claimed. Examples would include a 2D or 3D reference location, origin, anchor location etc. Additionally, creating maps from sensor data would fall within the scope of 3D modeling.
Claim Mapping:
Chaturvedi discloses a control method based on extended reality (Col. 1: “In the above example augmented reality UI, an augmented reality view of the image data can be provided on the screen and can include the changes to the first lighting profile.”), comprising:
obtaining a real environment image (“Sub-process 502, via the processor of the computing device, receives image data of a live camera view from the camera.”) and a spatial positional relationship of a real environment (“change in the relative position of the user with respect to the computing device.”), and
setting a virtual light source in the extended reality space (“When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”); and
rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source (“Further, in an example, a similar method from a scene or view capture to create light effect 312, as described above, is applicable to capture or build light profiles for lighting source 308. For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source”), and displaying the rendered extended reality space (“example implementations, images of items, such as, lighting items, are manipulated and rendered with changes. Such an example includes, storing, modifying, and/or other processing of the lighting items to change the shade on the item or the intensity of the light bulb within the item. Such changes are performed, in an example, on one of the content servers 418 or using other such entity”).
Chaturvedi does not explicitly disclose, but Boulton discloses in the same field of endeavor mapping the real environment to an extended reality space based on the real environment image and the spatial positional relationship (¶ 30: “a world-space-coordinate system may be mapped to a virtual-space-coordinate system so that the virtual position of the virtual object in the virtual space may be mapped to a corresponding world space position in the real world.”)
wherein mapping the real environment to the extended reality space based on the real environment image and the spatial positional relationship comprises: generating a two-dimensional or three-dimensional model corresponding to the real environment in an extended reality device to display the real environment in the extended reality space (Boulton, ¶ 32: “From three-dimensional representation 400, the augmented reality computing device generates a cube map 402 defining lighting conditions of the physical environment at the three-dimensional world-space position of the virtual object (i.e., lighting conditions of physical environment 104 at the location of virtual object 110). Because the augmented reality computing device already has the three-dimensional representation of the physical environment, the process of generating cube map 402 can be performed”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a mapping relationship as claimed.
One of ordinary skill in the art would have motivation to coordinate the positions of the virtual and real environment for the purpose of adding virtual elements, such as lighting (“a world-space-coordinate system may be mapped to a virtual-space-coordinate system so that the virtual position of the virtual object in the virtual space may be mapped to a corresponding world space position in the real world.”)(See Boulton, ¶ 30).. One of ordinary skill in the art would have had a reasonable expectation of success because Boulton’s techniques are applicable to lighting and Chaturvedi utilizes lighting relative to the real environment.
Claim 2
Chaturvedi discloses the method further comprises: obtaining morphological data of an environmental object in the real environment (“A structured lighting example includes projecting a regular pattern of light (e.g., IR light) to the objects and analyzing the changes in the reflected pattern due to the 3D features of the objects.”), initial lighting for environment (“Further, in an example, a similar method from a scene or view capture to create light effect 312, as described above, is applicable to capture or build light profiles for lighting source 308. For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source”), and/or initial lighting for environmental object (“The light information can be analyzed to determine a first lighting profile associated with the representation of the object. Position information associated with the object with respect to the camera can be determined.”).
Claim 3
Chaturvedi discloses wherein, the mapping the real environment to an extended reality space based on the spatial positional relationship comprises: mapping the real environment to the extended reality space based on one or more of the spatial positional relationship, the morphological data of the environmental object in the real environment, the initial lighting for environment, or the initial lighting for environmental object (“Further, in an example, a similar method from a scene or view capture to create light effect 312, as described above, is applicable to capture or build light profiles for lighting source 308. For example, the light profiles are obtained from a physical environment image capture of a physical light source associated with the virtual light source 308. Such a light profile, in an aspect, is obtained in one or more different physical ambient light situations for the light source 308. From this, an intensity model is built for each light source and is used to provide light intensities in various ambient lighting situations and angles for each of the corresponding light source. When the light source is added, as in FIG. 3B, the light profile associated with the light source 308, and stored in a database, is provided to overlay or to change the pixel properties of the surrounding areas of the added light source 308 and table 304, for instance.”)
Claim 4
Chaturvedi discloses, wherein, the method further comprises: calibrating an optical property of an environmental object mapped to the extended reality space (“The method and system herein, in an example, addresses the light surrounding each of these representations of the objects 124, 126, and any light interactions, such as reflections, etc.”)
Claim 5
Chaturvedi discloses wherein, the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises: rendering the light effect for the real environment mapped to the extended reality space based on the light source parameter and the optical property of the environmental object (“When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”)
Claim 6
Chaturvedi discloses wherein the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises:
rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source and reference data (“When the user adds a light source 308, the appearance of light 312 surrounding the representations of the objects 304, 306 are altered such that their textural shading, shadowing, and/or reflections correspond to of the inherent light profile of the added light source 308.”), wherein the reference data comprises one or more of:
a spatial dimension of the real environment (“The 2D and 3D image data, in an example, forms the basis of ambient lighting 302”), a positional relationship of environmental objects in the real environment (“The direction is in reference to the light direction for an object with respect to the camera of the computing device 320”), a material of an environmental object in the real environment (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”), or a user posture (“As the user tilts the device, the relative position of the light source can be tracked using the same detection approaches discussed elsewhere herein”).
Claim 7
Chaturvedi discloses wherein the light source parameter comprises one or more of the followings of the virtual light source: a light source position, a light source angle, a light source intensity, a hue, a focal length, a softness, and a shading degree (“The second lighting profile provides, in an example, at least intensity values and direction information for light projected from the light source.”)
Claim 8
Chaturvedi discloses wherein, the method further comprises: setting a virtual reflective object in the extended reality space (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”)
Claim 9
Chaturvedi does not explicitly disclose, but Boulton discloses wherein, the rendering a light effect for the real environment mapped to the extended reality space based on a light source parameter of the virtual light source comprises: determining a reflection of the reflective object based on an optical property of the reflective object; and rendering the real environment image mapped to the extended reality space based on the virtual light source and the reflection of the reflective object (Boulton, ¶ 45: “In some cases, the lighting effects that should be applied to a virtual object can change over time, either because lighting conditions of the physical environment change, or one or more properties of the virtual object (e.g., shape, size, position, color, and/or reflectivity) change over time.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider reflectivity as claimed.
One of ordinary skill in the art would have motivation to coordinate the positions of the virtual and real environment for the purpose of adding virtual elements, such as lighting and accounting for real world properties of materials in the scene (“In some cases, the lighting effects that should be applied to a virtual object can change over time, either because lighting conditions of the physical environment change, or one or more properties of the virtual object (e.g., shape, size, position, color, and/or reflectivity) change over time.”))(See Boulton, ¶ 45). This is further suggested by Chaturvedi (“Additionally, various other types of content or items can be utilized as well within the scope of the various embodiments—e.g., items providing reflection of ambient lighting for decorative purposes.”). One of ordinary skill in the art would have had a reasonable expectation of success because Boulton’s techniques are applicable to lighting and Chaturvedi utilizes lighting relative to the real environment.
Claim 10
Chaturvedi discloses wherein, the real environment comprises a photographic scene and a photographic object, and the virtual light source comprises a lighting source for the photographic object (Chaturvedi, Col. 3: “Accordingly, the systems and methods herein provide a richer experience in augmented reality or image/video modes on a screen, where additional representations of objects (e.g., tables and couches in photos or videos) to a captured view, image, or video is not merely a passive addition, but actively changes aspects of the representations of these objects. In an embodiment, the active changes can include changing a lighting surrounding one or more representations of objects using virtual lighting sources added to the captured view, image, or video.”)
Claim 11
Chaturvedi discloses wherein, the real environment comprises artwork (Chaturvedi, Col. 3: “FIGS. 1A-1C illustrate an example approach for providing a light source in a live camera view in accordance with an embodiment. FIG. 1A provides a view or scene 100 for capture using a camera or any image capture element (e.g., reference numeral 130 of FIG. 1B) of a computing device (e.g., reference numeral 128 of FIG. 1B). The scene includes objects, such as tables 104, 106, a couch and a picture frame.”), and the virtual light source comprises a light source of the artwork (Chaturvedi, Col. 3: “changing a lighting surrounding one or more representations of objects using virtual lighting sources added to the captured view, image, or video. This process allows a user to preview, for example, effects from a virtual light source added into a live camera view, an image, or a video on a screen.”).
Claim 12
Chaturvedi discloses wherein, the real environment comprises furniture (Chaturvedi, Col. 4: “A representation of an object may be the live camera view capture of furniture and human subjects in a field of view of the camera”), and the virtual light source comprises a lamp (Chaturvedi, Col. 4: “augmenting a light profile of a physical space represented in a live camera view with a light profile associated with a virtual lamp”).
Claim 14
The same teachings and rationales in claim 1 are applicable to claim 14, with Chatuvedi disclosing an electronic device comprising: at least one memory and at least one processor; wherein the at least one memory is configured to store program code, and the at least one processor is configured to call the program code stored in the at least one memory to execute a method comprising (Fig. 4: “In an example utilizing the network architecture 400, a computing device 402 includes a processor, a screen, a camera, and memory including instructions that the processor executes to perform processes described in this disclosure.”)
Claim 15
The same teachings and rationales in claim 2 are applicable to claim 15.
Claim 16
The same teachings and rationales in claim 3 are applicable to claim 16.
Claim 17
The same teachings and rationales in claim 4 are applicable to claim 17.
Claim 18
The same teachings and rationales in claim 5 are applicable to claim 18.
Claim 19
The same teachings and rationales in claim 6 are applicable to claim 19.
Claim 20
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Claim 20 as drafted recites a non-transitory computer-readable storage medium…
Because non-transitory explicitly excludes transitory propagating signals and the like, the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 1 are applicable to claim 20, with Chatuvedi disclosing a non-transitory computer readable medium as claimed (Col. 1: “Include computer-readable medium storing instructions that, when executed by a processor of the server, allow the server to perform its intended functions.”)
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chaturvedi (US Patent 10,210,664) in view of Boulton (US 2018/0182160) and Delamont (US 2020/0368616)
Claim 13
Chaturvedi does not disclose, but Delamont discloses wherein, the real environment comprises a real game scene, and the virtual light source comprises a virtual lighting prop (Delamont, ¶ 559, 579: “the game server 88 or host 89 may user external 3D projection mapping techniques together with its external projectors 96, to project lighting effects and augmented images of the IR Laser Beam, in which using same state information captured from the users IR Laser Gun Apparatus 47 these virtual images and lighting effects may be accurately augmented over real-world objects or surfaces.. a user's physical hand and arm movements this shall invoke the re-rendering of its surfaces to apply lighting and shading visual effects in real-time according to other real-game objects or virtual game objects world coordinates, directional vectors and relative directional lighting to that of the real world devices/game object world coordinates, directional vector and directional lighting.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a game scene.
One of ordinary skill in the art would have motivation to coordinate the positions of the virtual and real environment for the purpose of adding virtual elements, such as lighting to different real world objects. One of ordinary skill in the art would have had a reasonable expectation of success because Chaturvedi and Boulton’s techniques are applicable to lighting of any real objects and Delamont utilizes lighting relative to the real environment to improve the game experience. .
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights