Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 3-5, 13, 15, and 20 are amended. Claim 16 is cancelled. Claims 1-15 and 17-21 are currently under review
Response to Arguments
Applicant's arguments filed March 27, 2026 have been fully considered but they are not persuasive. Although we initially discussed that the amendments may potentially overcome the prior art rejection, however upon further consideration Zhang teaches the claim amendments in figure 8, paragraph 32, namely “the three-dimensional hit-test location may be converted into a two-dimensional hit test location on the surface of the application window. In other words, the X-Y-Z coordinates of the selected location within the coordinate space of the three-dimensional environment 56 are converted into X-Z coordinates of the hit-test location within a two-dimensional coordinate space on the two-dimensional surface of the first application window 206. Such two-dimensional coordinates are then injected into the application generating the application window 206”, and paragraph 46, namely “In response to determining that the user moves the second application window 208 outside the boundary, view management of the second application window may be transitioned from the operating system shell 46 to the three-dimensional (holographic) shell 60 corresponding to the three-dimensional environment 56”. Please see below rejection for further details.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 11, 13-14, and 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Singh et al. (Pub. No.: US 2024/0061657 A1) hereinafter referred to as Singh in view of Zhang et al. (Pub. No.: US 2019/0073109 A1) and in view of Ueno et al. (Pub. No.: US 2015/0312559 A1) hereinafter referred to as Ueno.
With respect to Claim 1, Singh teaches a method (fig. 8; ¶67, “the methods herein may be implemented as software instructions executed by a processor, suitably configured application specific integrated circuits (ASIC) or field programmable gate array (FPGA) modules, or any other convenient manner”; ¶114) comprising: at an electronic device (fig. 3, item 216; ¶90) having a processor (fig. 3, item 302; ¶90) and a display (fig. 3, item 304; ¶90): presenting an extended reality (XR) environment (¶90) comprising a virtual element (¶92, “the headset 216 is illustrated as a head-circumscribing VR headset, it may also be established by computerized smart glasses or another type of headset including other types of AR and MR headsets. For example, the headset may be established by an AR headset that may have a transparent display that is able to present 3D virtual objects/content”) and a cursor (¶26); obtaining image data associated with hand data corresponding to a three-dimensional (3D) motion of a hand in a XR environment, the hand data comprising hand gestures (¶48, “Interactions in 3D space (that may be translated to 2D interactions using App Space) may occur using any number of different 3D input modalities, including but not limited to gaze pointer, raycast, hand/arm gestures”; ¶91, “The camera 306 may also be used for gesture recognition to recognize gestures made by the user using their hand, arm, etc. consistent with present principles”; ¶97, “App Space may detect gaze, raycast, keyboard, and keypress events from any buttons on the head-mounted headset 400 itself or even other controller devices (such as 3D hand-held controllers) via the headset's own SDK for 3D rendering”); based on the hand data, operating in a first mode where 3D motion is converted to 2D planar motion on a 2D surface displayed within a 3D space of the XR environment (¶26, “App Space may intercept the 3D AR coordinates from a 3D cursor and convert them to 2D coordinates”; ¶93, “App Space may therefore render the 2D apps in a 3D spatial environment, as well as convert 3D coordinates in the 3D spatial coordinate system into 2D coordinates in the 2D coordinate system at runtime (and vice versa)”).
Singh does not explicitly mention that based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion; wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment.
Zhang teaches a method (figs. 13A to 13B; ¶65) comprising: at an electronic device (figs. 1-2, item 10) having a processor (fig. 1, item 32; ¶26) and a display (figs. 1-2, item 20 comprises item 21: display; ¶24): presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56); obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) cursor motion of a hand in the XR environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the spatial input signals is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶32; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals moving in the Z direction (fig. 8; fig. 13B, item 293; ¶46; ¶69); and based on the 3D spatial input signals moving in the Z direction (¶46; ¶69), modifying a mode of operation to a second mode during which the 3D motion of the spatial input signals is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment), wherein the 3D motion of the spatial input signals results in 3D movement of the virtual element in the z direction in the XR environment (fig. 8; fig. 13B, item 293; ¶46; ¶69).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Singh, such that spatial input signals are hand data comprising hand gestures that correspond to a cursor and image data, resulting in based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion, wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
Although Zhang does not explicitly indicate that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, Zhang teaches 2D planar motion with respect to movement in two directions without movement in a third direction.
Therefore it would have obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Singh, such that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, as there are finite number of movements in two directions without movement in a third direction, so as to provide alternative implementations.
With respect to Claim 2, claim 1 is incorporated, Singh does not mention wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion.
Zhang teaches a method (figs. 13A to 13B; ¶65) comprising: at an electronic device (figs. 1-2, item 10) having a processor (fig. 1, item 32; ¶26) and a display (figs. 1-2, item 20 comprises item 21: display; ¶24): presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56); obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) cursor motion of a hand in the XR environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the spatial input signals is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶32; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals moving in the Z direction (fig. 8; fig. 13B, item 293; ¶46; ¶69); and based on the 3D spatial input signals moving in the Z direction (¶46; ¶69), modifying a mode of operation to a second mode during which the 3D motion of the spatial input signals is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment), wherein the 3D motion of the spatial input signals results in 3D movement of the virtual element in the z direction in the XR environment (fig. 8; fig. 13B, item 293; ¶46; ¶69); wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion (¶32; ¶38).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Singh, wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
With respect to Claim 11, claim 1 is incorporated, Singh does not teach wherein: the first mode moves the cursor on a surface of the virtual element; and the second mode moves the virtual element within the XR environment while a position of the cursor on the surface of the virtual element is maintained.
Zhang teaches a method (figs. 13A to 13B; ¶65) comprising: at an electronic device (figs. 1-2, item 10) having a processor (fig. 1, item 32; ¶26) and a display (figs. 1-2, item 20 comprises item 21: display; ¶24): presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56); obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) pointer motion of a hand in a 3D environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the cursor is converted to 2D motion to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals (fig. 13B, item 293); and in response to said 3D spatial input signals, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment); wherein: the first mode moves the cursor on a surface of the virtual element (fig. 8, item the cursor 214 if on the surface of a virtual element 214 within 40; ¶46, “while displaying the two-dimensional pointer 214 hit-testing a second application window 208 located within the desktop window 40, a user selection of the application window may be received”); and the second mode moves the virtual element within the XR environment while a position of the cursor on the surface of the virtual element is maintained (fig. 9, item 208 is moved from item 40 to environment 56; ¶46, “the user 36 may move the second application window 208 outside the boundary of the desktop window 40 via interaction with the mouse 204. In response to determining that the user moves the second application window 208 outside the boundary, view management of the second application window may be transitioned from the operating system shell 46 to the three-dimensional (holographic) shell 60 corresponding to the three-dimensional environment 56”; ¶47).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Singh, wherein: the first mode moves the cursor on a surface of the virtual element; and the second mode moves the virtual element within the XR environment while a position of the cursor on the surface of the virtual element is maintained, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
With respect to Claim 13, Singh teaches an electronic device (fig. 3, item 216; ¶90) comprising: a non-transitory computer-readable storage medium (fig. 3, item 308; ¶12; ¶69-70; ¶92); and at least one processor (fig. 3, item 302; ¶90) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶66-67) that, when executed on the one or more processors, cause the electronic device to perform operations comprising: presenting an extended reality (XR) environment (¶90) comprising a virtual element (¶92, “the headset may be established by an AR headset that may have a transparent display that is able to present 3D virtual objects/content”) and a cursor (¶26), obtaining image data associated with hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (¶48, “Interactions in 3D space (that may be translated to 2D interactions using App Space) may occur using any number of different 3D input modalities, including but not limited to gaze pointer, raycast, hand/arm gestures”; ¶91, “The camera 306 may also be used for gesture recognition to recognize gestures made by the user using their hand, arm, etc. consistent with present principles”; ¶97, “App Space may detect gaze, raycast, keyboard, and keypress events from any buttons on the head-mounted headset 400 itself or even other controller devices (such as 3D hand-held controllers) via the headset's own SDK for 3D rendering”); based on the hand data, operating in a first mode where 3D motion is converted to 2D planar motion on a 2D surface displayed within a 3D space of the XR environment (¶26; ¶93, “App Space may therefore render the 2D apps in a 3D spatial environment, as well as convert 3D coordinates in the 3D spatial coordinate system into 2D coordinates in the 2D coordinate system at runtime (and vice versa)”).
Singh does not explicitly mention that based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion, wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment.
Zhang teaches an electronic device (figs. 1-2, items 10, 20, and 16) comprising: a non-transitory computer-readable storage medium (fig. 1, item 30; ¶26); and a processor (fig. 1, item 32; ¶26) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶26) that, when executed on the processor, causes the electronic device to perform operations comprising: presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56), obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) cursor motion of a hand in a XR environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the spatial input signals is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶32; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals moving in the Z direction (fig. 8; fig. 13B, item 293; ¶46; ¶69); and based on the 3D spatial input signals moving in the Z direction (¶46; ¶69), modifying a mode of operation to a second mode during which the 3D motion of the spatial input signals is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment), wherein the 3D motion of the spatial input signals results in 3D movement of the virtual element in the z direction in the XR environment (fig. 8; fig. 13B, item 293; ¶46; ¶69).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the electronic device of Singh, such that spatial input signals are hand data comprising hand gestures that correspond to a cursor and image data, resulting in based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion, wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
Although Zhang does not explicitly indicate that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, Zhang teaches 2D planar motion with respect to movement in two directions without movement in a third direction.
Therefore it would have obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the electronic device of Singh, such that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, as there are finite number of movements in two directions without movement in a third direction, so as to provide alternative implementations.
With respect to Claim 14, claim 13 is incorporated, Singh does not mention wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion.
Zhang teaches an electronic device (figs. 1-2, items 10, 20, and 16) comprising: a non-transitory computer-readable storage medium (fig. 1, item 30; ¶26); and a processor (fig. 1, item 32; ¶26) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶26) that, when executed on the one or more processors, cause the system to perform operations comprising: presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56), obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) pointer motion of a hand in a 3D environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the cursor is converted to 2D motion to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals (fig. 13B, item 293); and in response to said 3D spatial input signals, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment); wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion (¶32; ¶38).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the electronic device of Singh, wherein, based on said operating in the first mode, cursor movement, of the cursor, is limited to 2D motion, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
With respect to Claim 20, Singh teaches a non-transitory computer-readable storage medium (fig. 3, item 308; ¶12; ¶69-70; ¶92) storing program instructions (¶66-67) executable via at least one processor (fig. 3, item 302; ¶90), of an electronic device (fig. 3, item 216; ¶90), to perform operations comprising: presenting an extended reality (XR) environment (¶90) comprising a virtual element (¶92, “the headset may be established by an AR headset that may have a transparent display that is able to present 3D virtual objects/content”) and a cursor (¶26), obtaining image data associated with hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (¶48, “Interactions in 3D space (that may be translated to 2D interactions using App Space) may occur using any number of different 3D input modalities, including but not limited to gaze pointer, raycast, hand/arm gestures”; ¶91, “The camera 306 may also be used for gesture recognition to recognize gestures made by the user using their hand, arm, etc. consistent with present principles”; ¶97, “App Space may detect gaze, raycast, keyboard, and keypress events from any buttons on the head-mounted headset 400 itself or even other controller devices (such as 3D hand-held controllers) via the headset's own SDK for 3D rendering”); based on the hand data, operating in a first mode where 3D motion is converted to 2D planar motion on a 2D surface displayed within a 3D space of the XR environment (¶26; ¶93, “App Space may therefore render the 2D apps in a 3D spatial environment, as well as convert 3D coordinates in the 3D spatial coordinate system into 2D coordinates in the 2D coordinate system at runtime (and vice versa)”).
Singh does not explicitly mention that based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion; wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment.
Zhang teaches a non-transitory computer-readable storage medium (fig. 1, item 30; ¶26) storing program instructions (¶26) executable via a processor (fig. 1, item 32; ¶26), of an electronic device (figs. 1-2, items 10, 20, and 16), to perform operations comprising: presenting an extended reality (XR) environment (¶57) comprising a virtual element (fig. 11, item 240 and 244; ¶56) and a cursor (fig. 3, depicted as item 210 when outside a desktop window; fig. 11, depicted as item 214 when inside a desktop window or application window; ¶31, “In some examples, if the pointer 210 collides with virtual content in the three-dimensional environment 56, the pointer may be moved closer to the user 36 to overlap the virtual object or other content”; ¶56), obtaining spatial input signals associated with hand data corresponding to a three-dimensional (3D) cursor motion of a hand in the XR environment (¶30, “the three-dimensional pointer 210 may be controlled by other user input modalities, such as gaze detection using a targeting ray and/or gesture detection”); based on the spatial input signals, operating in a first mode where 3D motion of the spatial input signals is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment (fig. 7; fig. 13B, item 280-282; ¶32; ¶45, “the system may determine that the location of the three-dimensional pointer 210 moves from outside the boundary of the desktop window 40 to inside the window. In response, the translation of the spatial input signals may be changed from three-dimensional motion of the three-dimensional pointer 210 to two-dimensional motion of the two-dimensional pointer 214 within the desktop window 40”); detecting a 3D spatial input signals moving in the Z direction (fig. 8; fig. 13B, item 293; ¶46; ¶69); and based on the 3D spatial input signals moving in the Z direction (¶46; ¶69), modifying a mode of operation to a second mode during which the 3D motion of the spatial input signals is maintained without conversion to the 2D motion (fig. 13B, item 294; ¶41, “when the location of the two-dimensional pointer crosses the boundary area of the desktop window 40 into the surrounding three-dimensional environment 56, the two-dimensional pointer is replaced with the three-dimensional pointer 210 at the corresponding location”; ¶46; ¶65-66 – the 3D motion of the cursor is maintained since the cursor and the application (desktop window) are operating in a three-dimensional environment), wherein the 3D motion of the spatial input signals results in 3D movement of the virtual element in the z direction in the XR environment (fig. 8; fig. 13B, item 293; ¶46; ¶69).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the non-transitory computer-readable storage medium of Singh, such that spatial input signals are hand data comprising hand gestures that correspond to a cursor and image data, resulting in based on the hand data, operating in a first mode where 3D motion of the hand is converted to 2D planar motion with respect to movement in two directions without movement in a third direction to move the cursor on a 2D surface displayed within a 3D space of the XR environment; detecting a 3D hand gesture of the hand moving in the Z direction; and based on the 3D hand gesture of the hand moving in the z direction, modifying a mode of operation to a second mode during which the 3D motion of the hand is maintained without conversion to the 2D motion, wherein the 3D motion of the hand results in 3D movement of the virtual element in the z direction in the XR environment, as taught by Zhang so as to enable input to freely migrate between a desktop window and virtual space, thereby enabling a user to conveniently interact with desktop and non-desktop virtual content and also such that desktop applications also may be moved into and out from of a desktop window to provide continuum between an operating system shell and a holographic/three-dimensional shell displayed by an HMD device (¶23).
Although Zhang does not explicitly indicate that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, Zhang teaches 2D planar motion with respect to movement in two directions without movement in a third direction.
Therefore it would have obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the non-transitory computer-readable storage medium of Singh, such that 2D planar motion is with respect to movement in an X and Y direction without movement in a Z direction, as there are finite number of movements in two directions without movement in a third direction, so as to provide alternative implementations.
With respect to Claim 21, claim 1 is incorporated, Singh teaches wherein the 2D surface is an interface element (fig. 6, item 602; ¶110).
Claims 3-6, 8-10, 12, 15, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Singh and Zhang as applied to claims 1 and 13 above, and further in view of Cho et al. (Pub. No.: US 2019/0384420 A1) hereinafter referred to as Cho.
With respect to Claim 3, claim 1 is incorporated, Singh and Zhang combined do not explicitly mention wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 4, claim 3 is incorporated, Singh and Zhang combined do not explicitly mention wherein the 3D hand gesture of the hand moving in the Z direction determines that the cursor is on an object having an object type while a selection input is provided.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction determines that the cursor is on an object having an object type while a selection input is provided (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – the object is a window and the object type is an active window to be moved, click and move corresponds to a selection input).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the 3D hand gesture of the hand moving in the Z direction determines that the cursor is on an object having an object type while a selection input is provided, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 5, claim 1 is incorporated, Singh and Zhang combined do not explicitly mention wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 6, claim 5 is incorporated, Singh and Zhang combined do not explicitly mention wherein the method determines to provide the second mode while the user gesture is maintained.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); wherein the method determines to provide the second mode while the user gesture is maintained (¶97 or ¶45, hand data is always detected in 3D; fig. 17, item 1701 - depending on whether the control device is close to ground or not determines whether to provide the second mode or not, the second mode corresponding to items 1707 and 1709 – the user gesture is maintained corresponds to the control device not being close to ground while providing the hand data).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the method determines to provide the second mode while the user gesture is maintained, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 8, claim 5 is incorporated, Singh and Zhang combined do not explicitly mention further comprising moving both the cursor and the virtual object based on the 3D motion of the hand while the user gesture is maintained.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); further comprising moving both the cursor and the virtual object based on the 3D motion of the hand while the user gesture is maintained (fig. 19B; ¶146, “if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows. In other embodiments, the electronic device 101 may display the hand gesture or finger gesture of the user or the motion of the control device 102. For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102” -– the user gesture is maintained corresponds to the control device not being close to ground while providing the hand data).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in the method further comprising moving both the cursor and the virtual object based on the 3D motion of the hand while the user gesture is maintained, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 9, claim 5 is incorporated, Singh and Zhang combined do not mention further comprising determining to provide the first mode based on determining that the user gesture has been discontinued.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of UI element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); further comprising moving both the cursor and the virtual object based on the 3D motion of the hand while the user gesture is maintained (fig. 19B; ¶146, “if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows. In other embodiments, the electronic device 101 may display the hand gesture or finger gesture of the user or the motion of the control device 102. For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102” -– the user gesture is maintained corresponds to the control device not being close to ground while providing the hand data); further comprising determining to provide the first mode based on determining that the user gesture has been discontinued (fig. 17, item 1709 has completed and the process begins again at item 1701 and determined that the control device is close to ground -- the user gesture being discontinued corresponds to the control device being close to ground while providing the hand data).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in the method further comprising determining to provide the first mode based on determining that the user gesture has been discontinued, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 10, claim 1 is incorporated, Singh and Zhang combined do not explicitly mention further comprising enabling 3D movement of the virtual element using the second mode.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); further comprising enabling 3D movement of the virtual element using the second mode (fig. 19B; ¶146, “if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows. In other embodiments, the electronic device 101 may display the hand gesture or finger gesture of the user or the motion of the control device 102. For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in the method further comprising enabling 3D movement of the virtual element using the second mode, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 12, claim 1 is incorporated, Singh and Zhang combined do not teach further comprising moving the virtual object based on the 3D motion of the hand while the user gesture is maintained.
Cho teaches a method (fig. 20; ¶147) comprising: at an electronic device (figs. 1 and 6, item 101; ¶44) having a processor (fig. 1, item 120; ¶45) and a display (fig. 1, item 160; ¶48): presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) motion of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”); operating in first mode where 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); further comprising moving the virtual object based on the 3D motion of the hand while the user gesture is maintained (fig. 19B; ¶146, “if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows. In other embodiments, the electronic device 101 may display the hand gesture or finger gesture of the user or the motion of the control device 102. For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102” – the window is moved and the user gesture is maintained corresponds to the control device not being close to ground while providing the hand data).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in the method further comprising moving the virtual object based on the 3D motion of the hand while the user gesture is maintained, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 15, claim 13 is incorporated, Singh and Zhang combined do not explicitly mention wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of Ul element to be selected.
Cho teaches an electronic device (figs. 1 and 6, item 101; ¶44) comprising: a non-transitory computer-readable storage medium (fig. 1, item 130; ¶46); and one or more processors (fig. 1, item 120; ¶45) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶46) that, when executed on the one or more processors, cause the system to perform operations comprising: presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) movement of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”; ¶146, “the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); and operating in first mode where the 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of Ul element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined electronic device of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of Ul element to be selected, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 17, claim 13 is incorporated, Singh and Zhang combined do not explicitly mention wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture.
Cho teaches an electronic device (figs. 1 and 6, item 101; ¶44) comprising: a non-transitory computer-readable storage medium (fig. 1, item 130; ¶46); and one or more processors (fig. 1, item 120; ¶45) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶46) that, when executed on the one or more processors, cause the system to perform operations comprising: presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) movement of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”; ¶146, “the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); and operating in first mode where the 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of Ul element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined electronic device of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
With respect to Claim 18, claim 17 is incorporated, Singh and Zhang combined do not explicitly mention wherein the operations further comprise: a determination to provide the second mode while the user gesture is maintained.
Cho teaches an electronic device (figs. 1 and 6, item 101; ¶44) comprising: a non-transitory computer-readable storage medium (fig. 1, item 130; ¶46); and one or more processors (fig. 1, item 120; ¶45) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions (¶46) that, when executed on the one or more processors, cause the system to perform operations comprising: presenting an extended reality (XR) environment comprising a virtual element and a cursor (fig. 6; ¶87, “the electronic device 101 may form a viewport 610 in a real or virtual environment, and may output real external objects, virtual objects, or data information through the viewport 610 … In addition, the viewport 610 may refer to an area in which the pointer object is movable”); obtaining hand data corresponding to a three-dimensional (3D) movement of a hand in a 3D environment (via a control device – see figs. 1, 7, 8A/8B, 9 item 102; ¶50; ¶91; ¶97, “More specifically, the controller 770 may obtain a magnetic vector using the origin of the three-axis magnetic field (i.e., a reference point) generated in the electronic device 101 and information measured by the first sensor unit 710 (e.g., the intensity of current, the intensity of voltage, and the phase of a magnetic field signal), and may determine the coordinates of the control device 102 by means of a magnetic field formula” – since the hand data is comprises three axes, it therefore corresponds to 3D motion or ¶145, “If a camera of the electronic device 101 detects the hand shape of the user wearing the control devices 102, the three-dimensional pointer object may be displayed in the form of a user's hand or control device 102”; ¶146, “the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); and operating in first mode where the 3D motion of the cursor is converted to 2D motion (fig. 17, items 1701, 1703, and 1705; fig. 19, the electronic device operates in a first mode when it is determined that the control device is close to ground; ¶142; ¶143-144); detecting a 3D hand gesture of the hand moving in the Z direction (fig. 17, after item 1705 has ended, the process begins again and the 3D hand gesture of the hand moving in the Z direction is detected which comprises of the control device not being close to ground); and in response to said 3D hand gesture of the hand moving in the Z direction, modifying a mode of operation to a second mode where the 3D motion of the cursor is maintained without conversion to the 2D motion (fig. 17, items 1701, 1707, and 1709; ¶145-146); wherein the 3D hand gesture of the hand moving in the Z direction enables a particular type of Ul element to be selected (fig. 19B; ¶146, “in FIG. 19B, the electronic device 101 may move the three-dimensional pointer object 1903 up, down, left, right, forward, or backwards. In this case, if the electronic device 101 receives an input signal from the control device 102, the electronic device 101 may control the pointer object 1903 so as to click and move one of a plurality of windows” – “click” corresponds to selecting); wherein the 3D hand gesture of the hand moving in the Z direction comprises performing a user gesture and moving the hand in a z-direction (fig. 19B; ¶146, “For example, the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”); wherein the operations further comprise: a determination to provide the second mode while the user gesture is maintained (¶97 or ¶45, hand data is always detected in 3D; fig. 17, item 1701 - depending on whether the control device is close to ground or not determines whether to provide the second mode or not, the second mode corresponding to items 1707 and 1709 – the user gesture is maintained corresponds to the control device not being close to ground while providing the hand data).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined electronic device of Singh and Zhang, such that input via a control device is substituted with gesture input resulting in wherein the operations further comprise: a determination to provide the second mode while the user gesture is maintained, as taught by Cho, so as to allow seamless input in a variety of uses and applications.
Claims 7 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Singh, Zhang, and Cho as applied to claims 5 and 17 above, and further in view of Ma (Pub. No.: US 2021/0124417 A1).
With respect to Claim 7, claim 5 is incorporated, Although Cho teaches the electronic device may display the hand gesture or finger gesture of the user (¶146, “the electronic device 101 may display the rotational motion of hands, fingers, or the control device 102 using the three-dimensional direction coordinates (e.g., a roll angle, a pitch angle, and a yaw angle) received from the control device 102”), Singh, Zhang, and Cho combined do not explicitly mention wherein the user gesture comprises a pinch gesture.
Ma teaches a system (fig. 1A; ¶35) comprising: a processor (fig. 1A, item 106; ¶36) and a display (fig. 1A, item 108; ¶37); obtains hand data corresponding to a three-dimensional (3D) motion and detecting a 3D user input gesture (fig. 1A, via item 102; ¶36; ¶38-39; ¶101); wherein the 3D input criteria comprises performing a user gesture (¶33); wherein the user gesture comprises a pinch gesture (¶93; ¶95-99).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Singh, Zhang, and Cho, to incorporate the techniques of the system of Ma into the method of Cho, wherein the user gesture comprises a pinch gesture, as taught by Ma, so as to provide alternative types of hand data for performing a user gesture to perform mouse or text entry functionality (¶4).
With respect to Claim 19, claim 17 is incorporated, Singh, Zhang, and Cho combined do not mention wherein the user gesture comprises a pinch gesture.
Ma teaches a system (fig. 1A; ¶35) comprising: a processor (fig. 1A, item 106; ¶36) and a display (fig. 1A, item 108; ¶37); obtains hand data corresponding to a three-dimensional (3D) motion and detecting a 3D user input gesture (fig. 1A, via item 102; ¶36; ¶38-39; ¶101); wherein the 3D input criteria comprises performing a user gesture (¶33); wherein the user gesture comprises a pinch gesture (¶93; ¶95-99).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined electronic device of Singh, Zhang, and Cho, wherein the user gesture comprises a pinch gesture, as taught by Ma, so as to provide alternative types of hand data for performing a user gesture to perform mouse or text entry functionality (¶4).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hilliges et al. (Pub. No.: US 2012/0113223 A1) hereinafter referred to as Hilliges see figure 7 and ¶56.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONNA V Bocar whose telephone number is (571)272-0955. The examiner can normally be reached Monday - Friday 8:30am to 5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr A Awad can be reached at (571)272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONNA V Bocar/ Primary Examiner, Art Unit 2621
Prosecution Insights