DETAILED ACTION
This communication is in response to Application No. 18/198,828 originally filed 05/17/2023. The Request for Continued Examination and Amendment presented on 09/03/2025 which provides amendments to claims 1-5, 7-11, 13-19, 21, 23 are amended and claims 20, 22, 24 are cancelled is hereby acknowledged.
Currently claims 1-19, 21 and 23 are pending.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/03/2025 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 09/03/2025 have been fully considered but they are not persuasive. With respect to the prior art of Wang The Office respectfully disagrees. Wang defines virtual reality in paragraph [0031], as “A virtual reality (VR) environment refers to a simulated environment that is designed to be based entirely on computer-generated sensory inputs for one or more senses. A VR environment comprises a plurality of virtual objects with which a person may sense and/or interact. For example, computer-generated imagery of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with virtual objects in the VR environment through a simulation of the person's presence within the computer-generated environment, and/or through a simulation of a subset of the person's physical movements within the computer-generated environment.” The Office notes that Wang further states in paragraph [0039] that “…the head-mounted device 100 of a system 1 can be used in conjunction with an input device such as a keyboard 300. The user can observe the keyboard 300 or a representation thereof through the display 110 of the head-mounted device. The display can include an image and/or other representation of the keyboard 300, the keys 302, and/or the hands 20. It will be understood that the keyboard can be or include a traditional keyboard with keys for corresponding alpha and/or numeric inputs. It will be understood that other types of keyboards are contemplated, such as number pads, phone dial pads, security code entry pads, custom key pads, and the like.”. With respect to determining location, Wang teaches in paragraph [0043], “the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.” These paragraphs make clear that Wang is detecting motion and activation of individual keys and the combination of components (e.g. shown in Figure 4 ) would provide a signal during operation. The Office considers these elements expressly suggested based off of the disclosure of Wang and related paragraph [0042] which states “the keyboard 300 can be operatively connected to the head-mounted device 100, such that operation of the keys 302 generate signals that are transmitted from the keyboard 300 to the head-mounted device 100. facilitate one-way or two-way communication between the keyboard 300 and the head-mounted device”.
Regarding Applicants assertions with the word “signal” or pointing out its use in each paragraph in the prior art are not fully understood. However, an plain reading of the cited portions of [0043] with that of others portions, one of skill would readily understand that when paragraph [0061] which states “The head-mounted device 100 can include a camera 150 for capturing a view of an environment external to the head-mounted device 100. The camera 150 can include an optical sensor, such as a photodiode or a photodiode array. Additionally or alternatively, the camera 150 can include one or more of various types of optical sensors that are arranged in various configurations for detecting user inputs described herein.” provides a clear understanding that Wang teaches tracking motion and pressing of the keys. The device 100 in Wang utilizes various means of tracking (ie. camera, tracking device, etc.) in the environment to provide this determination. Contact and/or forces applied by a user can be detected by one or more components of the system 1. The operation of the keys 302 by the hands 20 of the user 10 can be communicated to the head-mounted device 100 in one or more of a variety of ways. (Wang [0041]) Paragraph [0045] makes clear again that Wang encompasses this idea stating, “It will be understood that combinations of the above can be provided such that operation of the keyboard 300 is detected based on multiple detections. For example, the head-mounted device 100 can operate independently and/or based on signals from a tracking device 200 and/or a keyboard 300. Where no signals from the keyboard 300 are required, the keyboard 300 can provide the structure of a conventional keyboard layout without any electronics that would generate signals in response to operation of the keys 302. As such, the keyboard 300 can provide a user with reference for the location of the keys 302 without requiring any electronics within the keyboard 300 that would respond to the operation of the keys 302. Additionally, the keyboard 300 can be compact, portable, and/or collapsible to facilitate transportation and storage thereof.” Thus Wang teaches multiple ways to for the computer-generated reality (CGR) environment to track and locate the fingers of the user and the interaction with the keys. Therefore, because Wang teaches providing a keyboard virtually in the display and tracking finger press/locations in relation to position on a key, Wang is expressly suggesting “pressing a physical key to determine the corresponding position of the virtual key within the 3D model of the input device in the virtual reality”.
Applicants remaining arguments in view of Berliner and claim 1 are moot as it is respectfully submitted the prior art of Wang fully teaches claim 1 as discussed above.
Claim Rejections - 35 USC § 112
Claim 19, 21, and 23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The claims recite “a virtual key” however the independent claims 1, 7, and 13 already recite the term “a virtual key” and it unclear if this is meant to reference the same “virtual key” or a different “virtual key”. To overcome this rejection The Office recommends changing “a” to “the” or specifying “first” or “second”.
Claim Rejections - 35 USC § 102
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 6, 7, 12, 13, and 18-24 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. U.S. Patent Application Publication No. 2020/0326847 A1 hereinafter Wang.
Consider Claim 1:
Wang discloses a method for projecting a three-dimensional (3D) model of an input device into a virtual reality, comprising: (Wang, See Abstract.)
obtaining a hand location of a hand model in the virtual reality, wherein the hand model corresponds to a hand of a user in real space; (Wang, [0039-0056], [0061], [0073], [0072], “For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment” [0043], “By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
receiving at least one input signal from a physical key in the input device, wherein the input device comprises a keyboard or a mouse, and wherein the input device is a physical input device in the real space and operated by the hand of the user, and (Wang, [0039-0056], [0061], [0040], “The user's hands 20 can be applied to the keys 302 of the keyboard 300 as an indication of text (e.g., characters, numbers, symbols, etc.) selected by the user 10. The operation of the keys 302 can produce text that is received, displayed, stored, and/or transmitted by the head-mounted device 100. The head-mounted device 100 can operate the camera 150 in a manner that captures one or more views of the keyboard 300 and/or the hands 20 within a field of view of the camera 150. The captured images can be produced on the display 110 of the head-mounted device 100 for observation by the user 10. As used herein, a camera is a device that can optically capture a view of an environment (e.g., within and/or outside the visible spectrum of light).”)
the at least one input signal is generated in response to the hand of the user activating the physical key in the physical input device, wherein the at least one input signal indicates activation of the physical key; (Wang, [0039-0056], [0061], [0043] By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
determining, based on the at least one input signal from the physical key activated by the hand of the user, a virtual key within the 3D model of the input device that corresponds to the physical key, obtaining a key position in the virtual reality of the virtual key within the 3D model of the input device, and (Wang, [0039-0056], [0061], [0045], “It will be understood that combinations of the above can be provided such that operation of the keyboard 300 is detected based on multiple detections. For example, the head-mounted device 100 can operate independently and/or based on signals from a tracking device 200 and/or a keyboard 300. Where no signals from the keyboard 300 are required, the keyboard 300 can provide the structure of a conventional keyboard layout without any electronics that would generate signals in response to operation of the keys 302. As such, the keyboard 300 can provide a user with reference for the location of the keys 302 without requiring any electronics within the keyboard 300 that would respond to the operation of the keys 302. Additionally, the keyboard 300 can be compact, portable, and/or collapsible to facilitate transportation and storage thereof.”)
determining a model location of the 3D model within the virtual reality based on the hand location and the key position, wherein the model location comprises a spatial position and an attitude of the 3D model; and (Wang, [0033], [0039-0056], [0061], [0048], “By further example, the head-mounted device 100 can detect operation of the input device 400 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the input device 400 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the regions of the input device 400 when pressed can be detected by the camera 150.”)
displaying the 3D model at the model location in a virtual reality scene to implement interaction between the input device and the virtual reality scene. (Wang, [0033], [0039-0056], [0061], [0072], “Referring now to FIG. 5, the display 110 of the head-mounted device 100 can provide a view of the keyboard 600, keys 602 of the keyboard, and/or a user's hands 620. One or more of the objects provided in the view of the display 110 can correspond to physical objects in an environment. For example, a camera of the head-mounted device 100 can capture a view of a physical keyboard, its physical keys, and/or the hands of the user. Based on the captured view, the display 110 can provide a display that includes images of the physical objects. Additionally or alternatively, the display 110 can provide a display of virtual objects that correspond to physical objects in the external environment. For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment. Additionally or alternatively, the display 110 can provide a display of virtual objects that do not correspond to physical objects in the external environment. For example, the keyboard 600 and/or the keys 602 can be rendered as virtual objects even when no physical keyboard or keys are present. By further example, a virtual keyboard and/or virtual keys can be displayed by the display 110 as if projected onto a surface even when no actual projection is provided on the surface. Such a virtual keyboard and/or virtual keys can be observable only through the display 110. It will be recognized that the view can include a view of physical objects and virtual objects.”)
Consider Claim 6:
Wang discloses the method according to claim 1, wherein after the 3D model is displayed at the model location in the virtual reality scene, the method further comprises: receiving the input signal from the input device to get an updated input signal; acquiring an updated hand location; and updating the 3D model in the virtual reality scene according to the updated input signal and the updated hand location. (Wang, [0090], “One or more characteristics of the key indicator 632 can be based on the distance between the user's hand 620 and the corresponding key 602. A key indicator 632 can be dynamically updated based on a distance between the user's hands and the corresponding key. For example, a key indicator 632 can grow larger as the user's hand 620 approaches a corresponding key 602, and/or the key indicator 632 can grow smaller as the user's hand 620 moves away from the corresponding key 602. Other characteristics, such as shape, color, aspect ratio, and the like can also be altered based on the user's hands 620. The key indicator 632 can include an animation. The key indicator 632 can be displayed as semi-transparent (e.g., semi-opaque), such that portions of the keyboard 600, the keys 602, and/or the user's hands 620 can be viewed through the key indicator 632. The key indicator can include outputs other than visual outputs. For example, the indicator can include audio and/or haptic feedback. For example, a key press can result in the head-mounted device outputting a sound that corresponds to a familiar sound of a key press on a physical keyboard. It will be recognized that other sounds can also be provided by the head-mounted device upon operation of one or more keys. Sounds can be global or location specific. For example, sound output in response to operation of keys at different locations can be output according to the location of the corresponding key. By further example, operation of keys on one side (e.g., left or right) of the keyboard can result in sounds being emitted from a speaker on a corresponding side (e.g., left or right) of the head-mounted device. This can provide the user with more spatial awareness of the actions being detected by the system.”)
Consider Claim 19:
Wang discloses the method according to claim 1, further comprising, after displaying the 3D model at the model location and in response to receiving a second input signal from the input device, displaying a manipulation effect of a virtual key corresponding to the second input signal in the 3D model. (Wang, [0033], [0039-0056], [0061], [0090], “The display of the head-mounted device can provide a view of the indicators over and/or near the keys of the keyboard. As shown in FIG. 11, key indicators 632 can be provided based on the operation of the keys 602. For example, when a key 602 is pressed by the user's hand 620, the key indicators 632 can be provided at or near the key 602 that has been pressed. Such a key indicator 632 can be used to provide a confirmation to a user that a key press has been received. For example, a depiction of a virtual keyboard can be displayed with virtual keys that correspond to the physical keys of a corresponding physical keyboard. When the physical keys are operated, the virtual keys are animated to travel as do the physical keys. The virtual keys can have at least some features that are different from the physical keys. For example, the virtual keys can be taller than the physical keys, and the animated travel can be more exaggerated than the travel of the physical keys. This provides the user with a more prominent visual feedback and confirmation that the action was received.”)
Consider Claim 7:
Wang discloses an electronic apparatus, comprising: (Wang, See Abstract.)
one or more processors; a memory coupled to the one or more processors; a display device that is configured to display a virtual reality scene; and a plurality of operations for projecting a virtual reality (VR) model stored in the memory that, when executed by the one or more processors, cause the electronic apparatus to perform the plurality of operations comprising: (Wang, [0058], “As shown in FIG. 4, the head-mounted device 100 can include a processor 170 with one or more processing units that include or are configured to access a memory 218 having instructions stored thereon. The instructions or computer programs may be configured to perform one or more of the operations or functions described with respect to the head-mounted device 100. The processor 170 can be implemented as any electronic device capable of processing, receiving, or transmitting data or instructions. For example, the processor 170 may include one or more of: a microprocessor, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or combinations of such devices. As described herein, the term “processor” is meant to encompass a single processor or processing unit, multiple processors, multiple processing units, or other suitably configured computing element or elements.”)
obtaining a hand location of a hand model in a virtual reality, wherein the hand model corresponds to a hand of a user in real space; (Wang, [0039-0056], [0061], [0073], [0072], “For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment” [0043], “By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
receiving at least one input signal from a physical key in an input device, wherein the input device comprises a keyboard or a mouse, and wherein the input device is a physical input device in the real space and operated by the hand of the user, and (Wang, [0039-0056], [0061], [0040], “The user's hands 20 can be applied to the keys 302 of the keyboard 300 as an indication of text (e.g., characters, numbers, symbols, etc.) selected by the user 10. The operation of the keys 302 can produce text that is received, displayed, stored, and/or transmitted by the head-mounted device 100. The head-mounted device 100 can operate the camera 150 in a manner that captures one or more views of the keyboard 300 and/or the hands 20 within a field of view of the camera 150. The captured images can be produced on the display 110 of the head-mounted device 100 for observation by the user 10. As used herein, a camera is a device that can optically capture a view of an environment (e.g., within and/or outside the visible spectrum of light).”)
the at least one input signal is generated in response to the hand of the user activating the physical key in the physical input device, wherein the at least one input signal indicates activation of the physical key; (Wang, [0039-0056], [0061], [0043] By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
determining, based on the at least one input signal from the physical key activated by the hand of the user, a virtual key within a three-dimensional (3D) model of the input device that corresponds to the physical key, obtaining a key position in the virtual reality of the virtual key within a three-dimensional (3D) model of the input device in the virtual reality, and (Wang, [0039-0056], [0061], [0045], “It will be understood that combinations of the above can be provided such that operation of the keyboard 300 is detected based on multiple detections. For example, the head-mounted device 100 can operate independently and/or based on signals from a tracking device 200 and/or a keyboard 300. Where no signals from the keyboard 300 are required, the keyboard 300 can provide the structure of a conventional keyboard layout without any electronics that would generate signals in response to operation of the keys 302. As such, the keyboard 300 can provide a user with reference for the location of the keys 302 without requiring any electronics within the keyboard 300 that would respond to the operation of the keys 302. Additionally, the keyboard 300 can be compact, portable, and/or collapsible to facilitate transportation and storage thereof.”)
determining a model location of the 3D model within the virtual reality based on the hand location and the key position, wherein the model location comprises a spatial position and an attitude of the 3D model; and (Wang, [0033], [0039-0056], [0061], [0048], “By further example, the head-mounted device 100 can detect operation of the input device 400 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the input device 400 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the regions of the input device 400 when pressed can be detected by the camera 150.”)
displaying the 3D model at the model location in a virtual reality scene to implement interaction between the input device and the virtual reality scene. (Wang, [0033], [0039-0056], [0061], [0072], “Referring now to FIG. 5, the display 110 of the head-mounted device 100 can provide a view of the keyboard 600, keys 602 of the keyboard, and/or a user's hands 620. One or more of the objects provided in the view of the display 110 can correspond to physical objects in an environment. For example, a camera of the head-mounted device 100 can capture a view of a physical keyboard, its physical keys, and/or the hands of the user. Based on the captured view, the display 110 can provide a display that includes images of the physical objects. Additionally or alternatively, the display 110 can provide a display of virtual objects that correspond to physical objects in the external environment. For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment. Additionally or alternatively, the display 110 can provide a display of virtual objects that do not correspond to physical objects in the external environment. For example, the keyboard 600 and/or the keys 602 can be rendered as virtual objects even when no physical keyboard or keys are present. By further example, a virtual keyboard and/or virtual keys can be displayed by the display 110 as if projected onto a surface even when no actual projection is provided on the surface. Such a virtual keyboard and/or virtual keys can be observable only through the display 110. It will be recognized that the view can include a view of physical objects and virtual objects.”)
Consider Claim 12:
Wang discloses the electronic apparatus according to claim 7, wherein after the 3D model is displayed at the model location in the virtual reality scene, the plurality of operations further comprises: receiving the input signal from the input device to get an updated input signal; acquiring an updated hand location; and updating the 3D model in the virtual reality scene according to the updated input signal and the updated hand location. (Wang, [0090], “One or more characteristics of the key indicator 632 can be based on the distance between the user's hand 620 and the corresponding key 602. A key indicator 632 can be dynamically updated based on a distance between the user's hands and the corresponding key. For example, a key indicator 632 can grow larger as the user's hand 620 approaches a corresponding key 602, and/or the key indicator 632 can grow smaller as the user's hand 620 moves away from the corresponding key 602. Other characteristics, such as shape, color, aspect ratio, and the like can also be altered based on the user's hands 620. The key indicator 632 can include an animation. The key indicator 632 can be displayed as semi-transparent (e.g., semi-opaque), such that portions of the keyboard 600, the keys 602, and/or the user's hands 620 can be viewed through the key indicator 632. The key indicator can include outputs other than visual outputs. For example, the indicator can include audio and/or haptic feedback. For example, a key press can result in the head-mounted device outputting a sound that corresponds to a familiar sound of a key press on a physical keyboard. It will be recognized that other sounds can also be provided by the head-mounted device upon operation of one or more keys. Sounds can be global or location specific. For example, sound output in response to operation of keys at different locations can be output according to the location of the corresponding key. By further example, operation of keys on one side (e.g., left or right) of the keyboard can result in sounds being emitted from a speaker on a corresponding side (e.g., left or right) of the head-mounted device. This can provide the user with more spatial awareness of the actions being detected by the system.”)
Consider Claim 21:
Wang discloses the electronic apparatus according to claim 7, wherein the plurality of operations for projecting a virtual reality (VR) model stored in the memory that, when executed by the one or more processors, further cause the electronic apparatus to perform the operations comprising, after displaying the 3D model at the model location and in response to receiving a second input signal from the input device, displaying a manipulation effect of a key corresponding to the second input signal in the 3D model. (Wang, [0090], “The display of the head-mounted device can provide a view of the indicators over and/or near the keys of the keyboard. As shown in FIG. 11, key indicators 632 can be provided based on the operation of the keys 602. For example, when a key 602 is pressed by the user's hand 620, the key indicators 632 can be provided at or near the key 602 that has been pressed. Such a key indicator 632 can be used to provide a confirmation to a user that a key press has been received. For example, a depiction of a virtual keyboard can be displayed with virtual keys that correspond to the physical keys of a corresponding physical keyboard. When the physical keys are operated, the virtual keys are animated to travel as do the physical keys. The virtual keys can have at least some features that are different from the physical keys. For example, the virtual keys can be taller than the physical keys, and the animated travel can be more exaggerated than the travel of the physical keys. This provides the user with a more prominent visual feedback and confirmation that the action was received.”)
Consider Claim 13:
Wang discloses an non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the electronic device to perform acts comprising: (Wang, [0058], See Abstract.)
obtaining a hand location of a hand model in a virtual reality, wherein the hand model corresponds to a hand of a user in real space; (Wang, [0039-0056], [0061], [0073], [0072], “For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment” [0043], “By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
receiving at least one input signal from a physical key in an input device, wherein the input device comprises a keyboard or a mouse, and wherein the input device is a physical input device in the real space and operated by the hand of the user, and (Wang, [0039-0056], [0061], [0040], “The user's hands 20 can be applied to the keys 302 of the keyboard 300 as an indication of text (e.g., characters, numbers, symbols, etc.) selected by the user 10. The operation of the keys 302 can produce text that is received, displayed, stored, and/or transmitted by the head-mounted device 100. The head-mounted device 100 can operate the camera 150 in a manner that captures one or more views of the keyboard 300 and/or the hands 20 within a field of view of the camera 150. The captured images can be produced on the display 110 of the head-mounted device 100 for observation by the user 10. As used herein, a camera is a device that can optically capture a view of an environment (e.g., within and/or outside the visible spectrum of light).”)
the at least one input signal is generated in response to the hand of the user activating the physical key in the physical input device, wherein the at least one input signal indicates activation of the physical key; (Wang, [0039-0056], [0061], [0043] By further example, the head-mounted device 100 can detect operation of the keys 302 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the keyboard 300 and the keys 302 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the keys 302 when pressed can be detected by the camera 150 or another device.”)
determining, based on the at least one input signal from the physical key activated by the hand of the user, a virtual key within a three-dimensional (3D) model of the input device that corresponds to the physical key, obtaining a key position in the virtual reality of the virtual key within the 3D model of the input device, and (Wang, [0039-0056], [0061], [0045], “It will be understood that combinations of the above can be provided such that operation of the keyboard 300 is detected based on multiple detections. For example, the head-mounted device 100 can operate independently and/or based on signals from a tracking device 200 and/or a keyboard 300. Where no signals from the keyboard 300 are required, the keyboard 300 can provide the structure of a conventional keyboard layout without any electronics that would generate signals in response to operation of the keys 302. As such, the keyboard 300 can provide a user with reference for the location of the keys 302 without requiring any electronics within the keyboard 300 that would respond to the operation of the keys 302. Additionally, the keyboard 300 can be compact, portable, and/or collapsible to facilitate transportation and storage thereof.”)
determining a model location of the 3D model within the virtual reality based on the hand location and the key position, wherein the model location comprises a spatial position and an attitude of the 3D model; and (Wang, [0033], [0039-0056], [0061], [0048], “By further example, the head-mounted device 100 can detect operation of the input device 400 with the user's hands 20 based on the views captured by the camera 150 of the head-mounted device 100. For example, the location of the hands 20 relative to the input device 400 can be observed optically. By further example, motion of the fingers of the hands 20 and or motion of the regions of the input device 400 when pressed can be detected by the camera 150.”)
displaying the 3D model at the model location in a virtual reality scene to implement interaction between the input device and the virtual reality scene. (Wang, [0033], [0039-0056], [0061], [0072], “Referring now to FIG. 5, the display 110 of the head-mounted device 100 can provide a view of the keyboard 600, keys 602 of the keyboard, and/or a user's hands 620. One or more of the objects provided in the view of the display 110 can correspond to physical objects in an environment. For example, a camera of the head-mounted device 100 can capture a view of a physical keyboard, its physical keys, and/or the hands of the user. Based on the captured view, the display 110 can provide a display that includes images of the physical objects. Additionally or alternatively, the display 110 can provide a display of virtual objects that correspond to physical objects in the external environment. For example, the keyboard 600, the keys 602, and/or the hands 620 can be rendered as virtual objects having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the external environment. Additionally or alternatively, the display 110 can provide a display of virtual objects that do not correspond to physical objects in the external environment. For example, the keyboard 600 and/or the keys 602 can be rendered as virtual objects even when no physical keyboard or keys are present. By further example, a virtual keyboard and/or virtual keys can be displayed by the display 110 as if projected onto a surface even when no actual projection is provided on the surface. Such a virtual keyboard and/or virtual keys can be observable only through the display 110. It will be recognized that the view can include a view of physical objects and virtual objects.”)
Consider Claim 18:
Wang discloses the non-transitory computer readable storage medium according to claim 13, wherein after the 3D model is displayed at the model location in the virtual reality scene, the electronic device is caused to perform acts further comprising: receiving the input signal from the input device to get an updated input signal; acquiring an updated hand location; and updating the 3D model in the virtual reality scene according to the updated input signal and the updated hand location. (Wang, [0090], “One or more characteristics of the key indicator 632 can be based on the distance between the user's hand 620 and the corresponding key 602. A key indicator 632 can be dynamically updated based on a distance between the user's hands and the corresponding key. For example, a key indicator 632 can grow larger as the user's hand 620 approaches a corresponding key 602, and/or the key indicator 632 can grow smaller as the user's hand 620 moves away from the corresponding key 602. Other characteristics, such as shape, color, aspect ratio, and the like can also be altered based on the user's hands 620. The key indicator 632 can include an animation. The key indicator 632 can be displayed as semi-transparent (e.g., semi-opaque), such that portions of the keyboard 600, the keys 602, and/or the user's hands 620 can be viewed through the key indicator 632. The key indicator can include outputs other than visual outputs. For example, the indicator can include audio and/or haptic feedback. For example, a key press can result in the head-mounted device outputting a sound that corresponds to a familiar sound of a key press on a physical keyboard. It will be recognized that other sounds can also be provided by the head-mounted device upon operation of one or more keys. Sounds can be global or location specific. For example, sound output in response to operation of keys at different locations can be output according to the location of the corresponding key. By further example, operation of keys on one side (e.g., left or right) of the keyboard can result in sounds being emitted from a speaker on a corresponding side (e.g., left or right) of the head-mounted device. This can provide the user with more spatial awareness of the actions being detected by the system.”)
Consider Claim 23:
Wang discloses the non-transitory computer readable storage medium according to claim 13, wherein the plurality of programs, when executed by the one or more processors, further cause the electronic device to perform acts comprising, after displaying the 3D model at the model location and in response to receiving a second input signal from the input device, displaying a manipulation effect of a virtual key corresponding to the second input signal in the 3D model. (Wang, [0090], “The display of the head-mounted device can provide a view of the indicators over and/or near the keys of the keyboard. As shown in FIG. 11, key indicators 632 can be provided based on the operation of the keys 602. For example, when a key 602 is pressed by the user's hand 620, the key indicators 632 can be provided at or near the key 602 that has been pressed. Such a key indicator 632 can be used to provide a confirmation to a user that a key press has been received. For example, a depiction of a virtual keyboard can be displayed with virtual keys that correspond to the physical keys of a corresponding physical keyboard. When the physical keys are operated, the virtual keys are animated to travel as do the physical keys. The virtual keys can have at least some features that are different from the physical keys. For example, the virtual keys can be taller than the physical keys, and the animated travel can be more exaggerated than the travel of the physical keys. This provides the user with a more prominent visual feedback and confirmation that the action was received.”)
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 2-5, 8-11, and 14-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. U.S. Patent Application Publication No. 2020/0326847 A1 as applied to claims 1 above, and further in view of Berliner et al. U.S. Patent Application Publication No. 2022/0256062 A1 hereinafter Berliner.
Consider Claim 2:
Wang discloses the method according to claim 1, and Wang discloses that hand locations are tracked and determined in 6D style environment (Wang, [0039-0056], [0063], [0044]) does not appear to provide the express details of wherein determining the model location based on the hand location and the key position comprises: obtaining the spatial position of the 3D model according to the key position within the 3D model and the spatial position within the hand location; determining an attitude of the 3D model according to the hand location of the hand model and the key position within the 3D model corresponding to a preset amount of input signals; and determining the model location of the 3D model in the virtual reality according to the spatial position and the attitude of the 3D model.
Berliner however teaches that it was a technique known to those having ordinary skill in the art before the effective filing date of the invention wherein determining the model location based on the hand location and the key position comprises: obtaining the spatial position of the 3D model according to the key position within the 3D model and a spatial position within the hand location; determining the attitude of the 3D model according to the hand location of the hand model and the key position within the 3D model corresponding to a preset amount of input signals; and determining the model location of the 3D model in the virtual reality according to the spatial position and the attitude of the 3D model. (Berliner, [0112], [0116], “In some embodiments, the extended reality appliance may receive digital signals, for example, from the input device. The term digital signals refers to a series of digital values that are discrete in time. The digital signals may represent, for example, sensor data, textual data, voice data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signals may be configured to cause the extended reality appliance to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signals may indicate a position and an angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signals may include an encoding of the position and angle in six degree-of-freedom coordinates (e.g., forward/back, up/down, left/right, yaw, pitch, and roll). In another embodiment, the digital signals may include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z), and an encoding of the angle as a vector originating from the encoded position. Specifically, the digital signals may indicate the orientation and an angle of the presented virtual content in an absolute coordinates of the environment, for example, by encoding yaw, pitch and roll of the virtual content with respect to a standard default angle. In another embodiment, the digital signals may indicate the orientation and the angle of the presented virtual content with respect to a viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding yaw, pitch, and roll of the virtual content with respect a direction corresponding to the viewpoint or to a direction corresponding to the other object. In another embodiment, such digital signals may include one or more projections of the virtual content, for example, in a format ready for presentation (e.g., image, video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signals may include a representation of virtual content, for example, by encoding objects in a three-dimensional array of voxels, in a polygon mesh, or in any other format in which virtual content may be presented.”)
It therefore would have been obvious to those having ordinary skill in the art before the effective filing date of the invention to use spatial positioning as this was a known technique for determining inputs in a virtual reality environment as this was a known technique in view of Berliner as one of ordinary skill in the art would readily recognize that enables the user to perceive and/or interact with a virtual reality environment. (Berliner, [0111])
Consider Claim 3:
Wang in view of Berliner discloses the method according to claim 2, wherein determining the attitude of the 3D model according to the hand location of the hand model and the key position within the 3D model corresponding to the preset amount of input signals comprises: determining key spatial positions of virtual keys corresponding to the preset amount of input signals according to spatial positions in the hand location; determining a spatial position of a plane in which the 3D model resides and an orientation of the 3D model according to the preset amount of hand locations and key positions of the preset amount of keys in the 3D model; and determining the attitude of the 3D model according to the spatial position of the plane and the orientation of the 3D model. (Wang, [0049], [0053], [0063], [0044], Berliner, [0116], “In some embodiments, the extended reality appliance may receive digital signals, for example, from the input device. The term digital signals refers to a series of digital values that are discrete in time. The digital signals may represent, for example, sensor data, textual data, voice data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signals may be configured to cause the extended reality appliance to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signals may indicate a position and an angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signals may include an encoding of the position and angle in six degree-of-freedom coordinates (e.g., forward/back, up/down, left/right, yaw, pitch, and roll). In another embodiment, the digital signals may include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z), and an encoding of the angle as a vector originating from the encoded position. Specifically, the digital signals may indicate the orientation and an angle of the presented virtual content in an absolute coordinates of the environment, for example, by encoding yaw, pitch and roll of the virtual content with respect to a standard default angle. In another embodiment, the digital signals may indicate the orientation and the angle of the presented virtual content with respect to a viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding yaw, pitch, and roll of the virtual content with respect a direction corresponding to the viewpoint or to a direction corresponding to the other object. In another embodiment, such digital signals may include one or more projections of the virtual content, for example, in a format ready for presentation (e.g., image, video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signals may include a representation of virtual content, for example, by encoding objects in a three-dimensional array of voxels, in a polygon mesh, or in any other format in which virtual content may be presented.”)
Consider Claim 4:
Wang in view of Berliner discloses the method according to claim 3, wherein determining the spatial position of the plane in which the 3D model resides and the orientation of the 3D model comprises: in response to determining that the preset amount is a first number: determining the spatial position of the plane in which the 3D model resides according to the spatial positions in the preset amount of hand locations of the hand model; and determining the orientation of the 3D model according to an orientation of a polygon formed by the first number of virtual keys, a spatial position and a position in the 3D model of each virtual key in the polygon. (Wang, [0049], [0053], [0063], [0044], Berliner, [0116], “In some embodiments, the extended reality appliance may receive digital signals, for example, from the input device. The term digital signals refers to a series of digital values that are discrete in time. The digital signals may represent, for example, sensor data, textual data, voice data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signals may be configured to cause the extended reality appliance to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signals may indicate a position and an angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signals may include an encoding of the position and angle in six degree-of-freedom coordinates (e.g., forward/back, up/down, left/right, yaw, pitch, and roll). In another embodiment, the digital signals may include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z), and an encod