Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 55, 57-61, 64-65, 73-74 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pekelny (US 2020/0026922) in view of Lee (US Patent 11,989,844)
Claim 55
Pekelny discloses a method comprising:
at a computer system in communication with one or more input devices and a display (Fig. 16; General PC hardware and HMD) generation component:
while displaying, via the display generation component, virtual content (202), wherein at least a portion of the virtual content obscures visibility of at least a portion of a physical environment of a user of the computer system (Pekelny, ¶ 40: “In the case of FIG. 2, assume that the user 104 has previously indicated that he wishes to be alerted to the existence of other people in the physical environment 102 when the user 104 is immersed in the virtual environment 202. This makes the presence of any person other than the user 104 an object-of-interest”),
PNG
media_image1.png
317
427
media_image1.png
Greyscale
detecting, via the one or more input devices, a passthrough visibility event (Pekelny, ¶ 40: “In the scenario of FIG. 1, the physical environment 102 does in fact include a person 108 in front of the user 104. The SPC detects this person 108 and then presents alert information 204 which notifies the user 104 of the existence of the other person 108”); and
in response to detecting the passthrough visibility event, replacing display, via the display generation component, of the at least the portion of the virtual content with presentation of a representation of a real-world object in the physical environment of the user (Pekelny, ¶ 40: “In the scenario of FIG. 1, the physical environment 102 does in fact include a person 108 in front of the user 104. The SPC detects this person 108 and then presents alert information 204 which notifies the user 104 of the existence of the other person 108…In this case, the alert information 204 may include a visual representation of the surface of the other person's body. Without limitation, in one example, the SPC can generate this kind of alert information using any three-dimensional reconstruction algorithm (e.g., the marching cubes algorithm) based on depth sensor readings provided by the VR device 106.”),
Pekelny does not explicitly disclose, but Lee in the same field of endeavor discloses:
PNG
media_image2.png
359
432
media_image2.png
Greyscale
in accordance with a determination that a state of the virtual content is a first state, presenting the representation of the real-world object with a first visual effect applied to the representation of the real-world object (e.g. above a threshold with more sharpness; Lee, Col. 15: “For example, the user may customize the boundary space 175 to include a starting threshold boundary 185, and an end boundary 195. Depending on the direction vector 155 of the user (e.g., as the user moves further away from or closer to the center of the boundary space 175), the computing system 110 (not shown) of the camera 105 may display on the VR display device 135 (not shown) the portion of the passthrough view with an increasing or decreasing size and sharpness. As an example, if the user is at the starting boundary 185, the size of the portion of the passthrough view may be relatively smaller than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively less sharp at the starting boundary 185 than it may be at the boundary 190. As another example, if the user is at the end boundary 195, the size of the portion of the passthrough view may be relatively larger than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively sharper at the end boundary 195 than it may be at the boundary 190.”); and
in accordance with a determination that the state of the virtual content is not the first state, presenting the representation of the real-world object without the first visual effect applied to the representation of the real-world object (e.g. below a threshold with no or less sharpness; Lee, Col. 15: “For example, the user may customize the boundary space 175 to include a starting threshold boundary 185, and an end boundary 195. Depending on the direction vector 155 of the user (e.g., as the user moves further away from or closer to the center of the boundary space 175), the computing system 110 (not shown) of the camera 105 may display on the VR display device 135 (not shown) the portion of the passthrough view with an increasing or decreasing size and sharpness. As an example, if the user is at the starting boundary 185, the size of the portion of the passthrough view may be relatively smaller than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively less sharp at the starting boundary 185 than it may be at the boundary 190. As another example, if the user is at the end boundary 195, the size of the portion of the passthrough view may be relatively larger than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively sharper at the end boundary 195 than it may be at the boundary 190.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a first and second state.
One of ordinary skill in the art would have motivation to provide a gradual adjustment or differing degrees of attention calling. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects.. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim 57
Pekelny discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, that the at least the portion of the virtual content has a spatial conflict with at least a portion of the real-world object, and presenting the representation of the real-world object includes presenting the at least the portion of the real-world object (e.g. a real object entering the space of the virtual environment;
PNG
media_image3.png
812
583
media_image3.png
Greyscale
¶ 40: “In the scenario of FIG. 1, the physical environment 102 does in fact include a person 108 in front of the user 104. The SPC detects this person 108 and then presents alert information 204 which notifies the user 104 of the existence of the other person 108. In this case, the alert information 204 may include a visual representation of the surface of the other person's body. Without limitation, in one example, the SPC can generate this kind of alert information using any three-dimensional reconstruction algorithm (e.g., the marching cubes algorithm) based on depth sensor readings provided by the VR device 106.”)
Claim 58
Pekelny discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, that the real-world object has moved to within a threshold distance of a location of the user in the physical environment (Pekelny, ¶ 64-65: “A depth camera system provides, at each moment, a depth image that reflects the distances between different points in the physical environment 102 and a reference point (typically associated with the location of the VR device 106 itself)… the VR device 106 may be preconfigured to detect walls in the physical environment 106 at a distance of 2 meters from the user 104, and then display alert information which represents the walls using pass-through video”)
Claim 59
Pekelny does not disclose, but Lee discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, that a viewpoint of the user is directed towards a boundary of the virtual content (Lee, Fig. 2A;
PNG
media_image4.png
198
243
media_image4.png
Greyscale
“As an example and not by way of limitation, sensors may determine the user is moving forward (e.g., in the direction of movement 125a) along the same direction as their field of view 120 (FIG. 2A).”)
wherein the real-world object is overlaid by the at least the portion of the virtual content, and wherein the at least the portion of the virtual content is adjacent to the boundary of the virtual content (Boundary is defined adjacent to real objects represented by virtual objects; Fig. 1C:
PNG
media_image5.png
780
537
media_image5.png
Greyscale
“The virtual boundary 115 may define or mark the edge of a safe area for the user to explore while the user is immersed in the VR environment 140. For example, in a room-scale VR environment (where the user can walk around a room during the VR experience) the virtual boundary 115 may correspond to real-world objects 145 (e.g., sofas, chairs, tables, walls, impediments, etc.) that the user wearing the VR display device 135 may like to avoid while immersed in the VR experience.”
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a boundary as claimed.
One of ordinary skill in the art would have motivation to provide an indication of the safe zone. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects relating to walls and other room specific features. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim 60
Pekelny does not disclose, but Lee discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, that a viewpoint of the user has moved more than a threshold distance from a location of the viewpoint of the user when the virtual content was first displayed (Lee, “The first threshold distance may be a predetermined distance (e.g., 1, 5, 10, etc. meters) from the virtual boundary 115. The first threshold distance may be determined by the user as well. As an example and not by way of limitation, in a room-scale VR setting, the computing system 110 may determine whether the user is approaching within a pre-determined distance of the virtual boundary 115. As another example and not by way of limitation, in a stationary VR setting, the first threshold distance may be when the user's head or hands approach the edge of the predetermined radius around the user (e.g., when the user's head or hands approaches a pre-determined 1 meter radius). Although this disclosure describes determining whether the user is approaching within a particular threshold distance of the virtual boundary in a particular manner, this disclosure contemplates determining whether the user is approaching within any suitable threshold distance of the virtual boundary in any suitable manner.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider a first and second state.
One of ordinary skill in the art would have motivation to provide a gradual adjustment or differing degrees of attention calling. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects.. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim 61
Pekelny discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, a user input corresponding to a request to cease to display an application associated with the virtual content (Pekelny, ¶ 14, 55: “the user (or the other person) makes a real-time command that triggers the presentation of the alert information to the user… For instance, the user 104 may issue a voice command, “Show floor now,”
Claim 64
Pekelny does not disclose, but Lee discloses wherein the virtual content is in the first state based at least in part on a determination that attention of the user is directed to the virtual content (e.g. Lee, Fig. 3A; “he passthrough view may be considered “directional” in that the area and position of the passthrough view may be based on the user's relative movement and field of view in the VR environment”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider user’s attention such as the direction of FOV.
One of ordinary skill in the art would have motivation to provide a gradual adjustment or differing degrees of attention calling. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects.. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim 65
Pekelny does not disclose, but Lee discloses wherein applying the first visual effect comprises reducing a visual prominence of the representation of the real-world object (e.g. adjusting sharpness; Lee, Col. 15: “For example, the user may customize the boundary space 175 to include a starting threshold boundary 185, and an end boundary 195. Depending on the direction vector 155 of the user (e.g., as the user moves further away from or closer to the center of the boundary space 175), the computing system 110 (not shown) of the camera 105 may display on the VR display device 135 (not shown) the portion of the passthrough view with an increasing or decreasing size and sharpness. As an example, if the user is at the starting boundary 185, the size of the portion of the passthrough view may be relatively smaller than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively less sharp at the starting boundary 185 than it may be at the boundary 190. As another example, if the user is at the end boundary 195, the size of the portion of the passthrough view may be relatively larger than it may be if the user was at the boundary 190. The sharpness of the portion of the passthrough view may be relatively sharper at the end boundary 195 than it may be at the boundary 190.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider reducing visual prominence.
One of ordinary skill in the art would have motivation to provide a gradual adjustment or differing degrees of attention calling. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects.. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim 73
The same teachings and rationales in claim 55 are applicable to claim 73, with Pekelny disclosing computer system that is in communication with a display generation component and one or more input devices, the computer system comprising:
PNG
media_image6.png
835
588
media_image6.png
Greyscale
one or more processors (1614);
memory (1606); and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for (1608; 1624)
Claim 74
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Claim 74 as drafted recites A non-transitory computer readable storage medium…
The broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 55 are applicable to claim 74 with Pekelny disclosing a computer readable storage medium (e.g. 1606)
Claim(s) 56 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pekelny (US 2020/0026922) in view of Lee (US Patent 11,989,844) and Labeau (US Patent 12,223,104)
Claim 56
Pekelny does not disclose, but Labeau discloses wherein detecting the passthrough visibility event comprises detecting, via the one or more input devices, that a portion of the user has moved into the at least the portion of the physical environment, and presenting the representation of the real-world object includes presenting a representation of the portion of the user (LaBeau, Fig. 6
PNG
media_image7.png
419
516
media_image7.png
Greyscale
“FIG. 6 is a conceptual diagram illustrating an example 600 of halo partial video passthrough being overlaid on a virtual environment 610 in some implementations employing the disclosed technology. In the example 600, a hand 615 of the user passes through a virtual wall 620 of the virtual environment 610 below a defined threshold height 625. When the hand 615 passes through the virtual wall 620 of the virtual environment below the defined threshold height 625, a hand passthrough event trigger is detected as described at block 502. Then, the process 500 determines what will be included in the hand passthrough video feed, such as the size of the halo of the passthrough 605, as described at block 504. In example 600, the size of the halo is defined as an area offset from both A) the contours of the hand 615 and B) an object 630 that the hand 615 is within a threshold distance of. The process 500 aligns the hand passthrough video feed within the virtual environment by receiving data from one or more cameras and/or sensors as described with regards to block 506 of Figure. The process 500 then displays object 630, such as a coffee cup, in the user's real-world environment overlaid on the virtual environment 610 in the halo of the partial passthrough 605.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider the claimed passthrough event.
One of ordinary skill in the art would have motivation to indicate context specific objects such as when a user wishes to interact. This is further suggested by Pekelny which considers individually setting distance metrics mapped to different visual effects.. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider pass through in the same context and could be readily combined.
Claim(s) 62, 63, 66-70 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pekelny (US 2020/0026922) in view of Lee (US Patent 11,989,844) and McHugh (US 2019/0279407)
Claim 62
Pekelny does not disclose, but McHugh discloses wherein in response to detecting the passthrough visibility event and in accordance with a determination that the state of the virtual content is a second state, wherein in the second state the virtual content comprises an application window, the representation of the real-world object is presented without a visual effect applied to the representation of the real-world object based on the state of the virtual content being the second state (e.g. adjustment of RGB values; McHugh ¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 63
Pekelny does not disclose, but McHugh discloses wherein the virtual content is in a second state when the virtual content comprises a user interface for entering information associated withan application, the user interface displayed concurrently with an application window associated with the application, and in response to detecting the passthrough visibility event and in accordance with a determination that the virtual content is in the second state, the representation of the real-world object is presented with a second visual effect different from the first visual effect ((e.g. adjustment of RGB values; McHugh¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 66
Pekelny does not disclose, but McHugh discloses wherein the first visual effect comprises a tint effect applied to the representation of the real-world object (e.g. adjustment of color in RGB; (e.g. adjustment of RGB values; McHugh ¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 67
Pekelny does not disclose, but McHugh discloses wherein the virtual content includes virtual media content and the tint effect is associated with one or more colors included in the virtual media content (e.g. adjustment of RGB values; McHugh ¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 68
Pekelny does not disclose, but McHugh discloses wherein the virtual content is associated with an application and the tint effect is selected based on the application associated with the virtual content (e.g. adjustment based on mapping 720;
PNG
media_image8.png
451
574
media_image8.png
Greyscale
McHugh ¶ 102: “The web 700a depicts of priority zero 710a, priority one 712a, priority two 714a, priority three 716a, priority four 718a, and priority n 720a. The priority n 720a can represent any number of additional priority levels. The priority system 700b depicts a similar a priority system of the web 700a of FIG. 7A. In particular, priority system 700b includes objects that are assigned various priorities from the priority web 700a. For example, the priority zero 710b is a different instance of the priority zero 710a of FIG. 7A. In another example, the priority one 712b is a different instances of the priority one 712a of FIG. 7A. In another example, the priority three 716b and priority three 716ci are different instances of the priority three 716a of FIG. 7A. In another example, the priority four 718b is a different instances of the priority four 718a of FIG. 7A..”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 69
Pekelny does not disclose, but McHugh discloses wherein the first visual effect comprises a change in saturation of the representation of the real-world object (e.g. adjustment of RGB values; McHugh ¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Claim 70
Pekelny does not disclose, but McHugh discloses wherein the virtual content includes an application window and a virtual environment, and the first visual effect is based at least in part on the application window and on the virtual environment (e.g. adjustment of RGB values; McHugh ¶ 126, 170: “The virtual objects of the AR environment 800b are a calendar event 802a, a flower 804a, and a messaging application 806a…. In block 1610, the rendering engine 340 receives a color associated with each pixel of a tagged element of a virtual object. Each virtual object can have tagged portions and non-tagged portions. A tagged portion is associated with a portion of the virtual object that the rendering engine 340 is to modify. A non-tagged portion is associated with a portion of the virtual object that the rendering engine 340 does not modify.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to modify the color as claimed
One of ordinary skill in the art would have motivation to use color as another means of calling the user’s attention. One of ordinary skill in the art would have had a reasonable expectation of success because Pekelny considers mapping different callout types to object actions.
Allowable Subject Matter
Claim(s) 71, 72 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim(s) 71, Lee considers an environment specific priority system:
PNG
media_image8.png
451
574
media_image8.png
Greyscale
McHugh would not suggest the modification of applying different tint to the real object based on which virtual environment.
Regarding claim(s) 72, parent claim 71 contains allowable subject matter.
Additional Prior Art
Additional prior art relevant to Applicant’s disclosure but not relied upon:
Mallinson (US 2016/0025978) considers selective see through (See Fig. 4)
PNG
media_image9.png
362
450
media_image9.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights