DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed on 06/11/2025. Claims 1, 4-10, 12-14 and 17-23 are pending in this application.
Applicant Response
In Applicant’s response dated 06/11/2025, Applicant amends claims 1, 6, 12, and 23 and argued against all objections and rejections previously set forth in the Office Action dated 03/14/2025.
Examiner Comments
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. /
6. Claims 1, 4-10, 12-14 and 17-23 are rejected under 35 U.S.C. 103 as being unpatentable over WILLIAMS (Pub. No.: US 20210271370 A1, Pub. Date 2021-09-02) in view of Algreatly (Pub. No.: US 20080062126 A1, Pub. Date 2008-03-13) in further view of Ofstad (Pub. No.: US 20150185873 A1, Pub. Date 2015-07-02) in further view of ZHOU (Pub. No.: US 20230143010 A1, Pub. Date 2023-05-11) in further view of Day (Pub. No.: US 20190235729 A1, Pub. Date 2019-08-01)
Regarding independent Claim 1,
Williams teaches the method comprising:
generating a three-dimensional (“3D”) environment comprising at least a two-dimensional (“2D”) user interface (“UI”) element and a 3D UI element (see Williams: Fig.9A-9C, [0128], “The screen 902 also displays 2D elements 904, 906, 908 that are graphics (e.g., object images) for viewing by the user. The screen 902 further displays a cursor 910, which is controllable by the user via a user input device. … [0130], “the processing unit 130 is configured to provide graphics (e.g., a displayed object) based on the accessed 3D model for display by the screen 902. For example, as shown in FIG. 9C, the graphics may be an object 950 that is rendered based on the 3D model. In some embodiments, the object 950 may be a 3D version of the 2D object 906 selected by the user. In some embodiments, the object 950 may be generated by the processing unit 130 based on the 3D model. Also, in some embodiments, the object 950 generated based on the 3D model may be the 3D model itself.”)
attaching a pointer to the 2D UI element in response to the pointer moving within a threshold distance of the 2D UI element (see Williams: Fig.9A-9C, [0129], “In response to the cursor 910 being in an associated position with respect to the 2D element 906, the processing unit 130 then provides an indicator 930 (e.g., generated by the graphic generator 430) for display by the screen 902. The indicator 930 indicates to the user that the 2D element 906 has an associated 3D model that may be retrieved.”), wherein attaching the pointer to the 2D UI element comprises presenting the pointer with a first flattened 2D shape that has a selection tip (see Williams: Fig.9A-9C, [0129], “the user may operate the user input device to move the cursor 910 so that the cursor 910 is pointing at 2D element 906 (which is an image of a mobile phone in the illustrated example). In response to the cursor 910 being in an associated position with respect to the 2D element 906, the processing unit 130 then provides an indicator 930 (e.g., generated by the graphic generator 430) for display by the screen 902.”)
WILLIAMS does not explicitly teach or suggest the method wherein:
performing a first mapping of inputs that control a movement of the pointer to coordinates of the 2D UI element that are presented about a specific 2D plane in the 3D environment in response to attaching the pointer to the 2D UI element instead of the 3D UI element, wherein performing the first mapping comprises:
receiving first inputs hat are performed in a 3D space and that are defined with three dimensional values as the first inputs are issued from an input device; and
moving the first flattened 2D shape with the selection tip continually contacting the 2D UI element about the specific 2D plane by:
tracking one or more of the three dimensional values that move the first flattened 2D shape away from the specific 2D plane at which the 2D UI element is presented in the 3D environment less than a threshold amount; and
changing the one or more of the three dimensional values issued from the input device to retain the first flattened 2D shape on the specific 2D plane;
selecting a first set of pixels of the 2D UI element that are contacted by the selection tip based on the first mapping of the three dimensional values of the first inputs to a first set of the coordinates of the 2D UI element about the specific 2D plane that are associated with the first set of pixels;
attaching the pointer to the 3D UI element in response to the pointer moving within the threshold distance of the 3D UI element in the 3D environment, wherein attaching the pointer to the 3D UI element comprises presenting the pointer with a second volumetric shape having a proximal tip and a distal end;
performing a second mapping of the inputs that control the movement of the pointer to coordinates of the 3D UI element in response to attaching the pointer to the 3D UI element instead of the 2D UI element, wherein the second mapping is different than the first mapping and performing the second mapping comprises:
receiving second inputs that are performed in the 3D space and that are defined with different three dimensional values as the second inputs are issued from the input device; and
moving the second volumetric shape such that with the proximal tip tracks tracking a 3D curvature of the 3D UI element and with the distal end extending away and off the 3D curvature by:
tracking two or more of the different three dimensional values of the second inputs that move the second volumetric shape off the 3D curvature of the 3D UI element by less than the threshold amount; and
changing the two or more of the different three dimensional values to match retain the proximal tip on 3D curvature of the 3D UI element; and
selecting a second set of pixels of the 3D UI element that are contacted by the proximal tip based on the second mapping of the different three dimensional values of the second inputs a second set of coordinates of the 3D UI element that are associated with the second set of pixels.
However, Algreatly teaches the method wherein:
performing a first mapping of inputs that control a movement of the pointer to coordinates of the 2D UI element that are presented about a specific 2D plane in the 3D environment in response to attaching the pointer to the 2D UI element instead of the 3D UI element (see Algreatly: Fig.5.2-5.3, [0044], “providing a movement along the positive x-axis, moves the pointer and the virtual camera along the positive x-axis of the hand-held device's display as illustrated in FIG. 5.2. Providing a movement along the negative x-axis, moves the pointer and the virtual camera along the negative x-axis of the hand-held device's display as illustrated in FIG. 5.3.”), wherein performing the first mapping comprises:
performing a second mapping of the inputs that control the movement of the pointer to coordinates of the 3D UI element in response to attaching the pointer to the 3D UI element instead of the 2D UI element, wherein the second mapping is different than the first mapping (see Algreatly: Fig.6, [0052], “For example, if the endpoint of the pointer is intersected with the cube in node (0, 0, 0) and the pointer is rotated clockwise about the y-axis of the hand-held device's display, then the endpoint of the pointer will be moved parallel to the xy-plane of the cube, respectively, on nodes (1, 0, 0), (2, 0, 0), (3, 0, 0), (4, 0, 0), (5, 0, 0), (6, 0, 0), (6, 1, 0), (6, 2, 0), and (6, 3, 0). Also, if the pointer is rotated counter-clockwise about the x-axis of the hand-held device's display then the endpoint of the pointer will be moved on the yz-plane of the cube, respectively, on nodes (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 1, 6), (0, 2, 6), and (0, 3, 6).”)
Because both WILLIAMS and Algreatly are in the same/similar field of endeavor of control and interaction with two-dimensional (“2D”) and three-dimensional (“3D”) objects in 3D space or environment accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the method that map the movement of a pointer or cursor 3D coordinates of the 3D UI element or 2D coordinates of the 2D UI element when the pointer is attaches attached to a 3D UI element or 2D UI element as taught by Algreatly. One would have been motivated to make such a combination in order to provide users an efficient, error free and precise cursor manipulation of virtual objects to simplify and enhance user the experience with the virtual/augmented reality environment. (See Algreatly [0006])
WILLIAMS and Algreatly does not teach the system wherein:
performing the first mapping comprises:
receiving first inputs hat are performed in a 3D space and that are defined with three dimensional values as the first inputs are issued from an input device; and
moving the first flattened 2D shape with the selection tip continually contacting the 2D UI element about the specific 2D plane by:
tracking one or more of the three dimensional values that move the first flattened 2D shape away from the specific 2D plane at which the 2D UI element is presented in the 3D environment less than a threshold amount; and
changing the one or more of the three dimensional values issued from the input device to retain the first flattened 2D shape on the specific 2D plane;
selecting a first set of pixels of the 2D UI element that are contacted by the selection tip based on the first mapping of the three dimensional values of the first inputs to a first set of the coordinates of the 2D UI element about the specific 2D plane that are associated with the first set of pixels;
attaching the pointer to the 3D UI element in response to the pointer moving within the threshold distance of the 3D UI element in the 3D environment, wherein attaching the pointer to the 3D UI element comprises presenting the pointer with a second volumetric shape having a proximal tip and a distal end.”), wherein the second mapping is different than the first mapping and
performing the second mapping comprises:
receiving second inputs as the second inputs are issued from the input device; and
moving the second volumetric shape such that the proximal tip tracks 3D curvature of the 3D UI element with the distal end extending away and off the 3D curvature as a result of continually adjusting first and second dimensional values of the second inputs that differ from the coordinates of the 3D UI element in two of the three dimensions by less than the threshold amount, to match the coordinates of the 3D UI element in the two dimensions;
selecting a second set of pixels of the 3D UI element that are contacted by the proximal tip based on the second mapping of the different three dimensional values of the second inputs a second set of coordinates of the 3D UI element that are associated with the second set of pixels.
However, Ofstad teaches the method wherein:
performing the first mapping (see Ofstad: Fig.2B, [0029], “the object data structure 279 may include data that defines an (x, y) coordinate location of an object 279a, such as object 110A of FIG. 1A”), comprises:
receiving first inputs that are performed in a 3D space and that are defined with three dimensional values as the first inputs are issued from an input device (see Ofstad: Fig.1C, [0023], “3D cursor 115C subsequent to automatic movement of the 3D cursor 115B of FIG. 1B to the object and subsequent to receiving an indication of a user selection of the object 110B of FIG. 1B.”); and
moving the first flattened 2D shape with the selection tip continually contacting the 2D UI element about the specific 2D plane (see Ofstad: Fig.1A-1C, [0018], “change in the geometric shape of the 3D cursor may indicate automatic movement of the 3D cursor. For example, the 3D cursor geometric shape may change from a first geometric shape, such as a pancake shape, to a second geometric shape, such as a square shape, a rectangle shape, a circular shape, an ovular shape, etc.”), by: […]
selecting a first set of pixels of the 2D UI element that are contacted by the selection tip based on the first mapping of the three dimensional values of the first inputs to a first set of the coordinates of the 2D UI element about the specific 2D plane that are associated with the first set of pixels (see Ofstad: Fig.2B, [0029], “The object data structure 279 may further include data that represents various imagery of objects 279c, such as various perspective views of the objects. The object data structure 279 may yet further include data 279d that defines colors and shading of various objects. The object data structure 279 may include (x, y, z) coordinate data that defines at least a portion of an object in addition to, or in lieu of (x, y) coordinate data.”),
attaching the pointer to the 3D UI element in response to the pointer moving within the threshold distance of the 3D UI element in the 3D environment (see Ofstad: Fig.5, [0033], “The processor 225 may determine a 3D cursor location based on the 3D cursor (x, y) coordinate location data 278a and determine an object location based on the object (x, y) coordinate location data 279a. When the processor 225 determines that the 3D cursor location is not less than a threshold distance from an object location in block 520, the flow returns to block 515. When the processor 225 determines that the 3D cursor is less than a threshold distance from an object in block 520, the processor 225 automatically modifies geometric shape data of the 3D cursor to at least partially surround the object in block 525.”), [… ] wherein the second mapping is different than the first mapping (see Ofstad: Fig.2B, [0029], “The object data structure 279 may yet further include data 279d that defines colors and shading of various objects. The object data structure 279 may include (x, y, z) coordinate data that defines at least a portion of an object in addition to, or in lieu of (x, y) coordinate data.”), performing the second mapping comprises:
receiving second inputs as the second inputs are issued from the input device (see Ofstad: Fig.1A-1C, [0018], “change in the geometric shape of the 3D cursor may indicate automatic movement of the 3D cursor. For example, the 3D cursor geometric shape may change from a first geometric shape, such as a pancake shape, to a second geometric shape, such as a square shape, a rectangle shape, a circular shape, an ovular shape, etc.).” ); and
moving the second volumetric shape such that the proximal tip tracks 3D curvature of the 3D UI element with the distal end extending away and off the 3D curvature as a result of continually adjusting first and second dimensional values of the second inputs that differ from the coordinates of the 3D UI element in two of the three dimensions by less than the threshold amount, to match the coordinates of the 3D UI element in the two dimensions (see Ofstad: Fig.2B, Fig.5, [0033], “The processor 225 determines (x, y coordinate) locations for each object within the scene in block 510. The processor 225 retrieves a 3D cursor (x, y coordinate) location from a 3D cursor positioning device in block 515. The processor 225 makes a determination as to whether the 3D cursor location is less than a threshold distance from an object location in block 520. As described with reference to FIG. 2B, the 3D scene data structure 277 may include (x, y) coordinates reference data 277a. The processor 225 may establish a common (x, y) coordinate reference for both the 3D cursor (x, y) coordinate location data 278a and the object (x, y) coordinate location data 279a based on the 3D scene (x, y) coordinates reference data 277a.”);
selecting a second set of pixels of the 3D UI element that are contacted by the proximal tip based on the second mapping of the different three dimensional values of the second inputs a second set of coordinates of the 3D UI element that are associated with the second set of pixels (see Ofstad: Fig.2B, Fig.5, [0033], “The processor 225 may determine a 3D cursor location based on the 3D cursor (x, y) coordinate location data 278a and determine an object location based on the object (x, y) coordinate location data 279a. When the processor 225 determines that the 3D cursor location is not less than a threshold distance from an object location in block 520, the flow returns to block 515. When the processor 225 determines that the 3D cursor is less than a threshold distance from an object in block 520, the processor 225 automatically modifies geometric shape data of the 3D cursor to at least partially surround the object in block 525. Subsequent to the processor 225 automatically modifying geometric shape data of the 3D cursor in block 525, the processor 225 updates a display based on point of interest data. In some embodiments, the processor 225 updates the display based on data representative of a best available view of the corresponding object in block 530.”)
Because WILLIAMS, Algreatly and Ofstad are in the same/similar field of endeavor of 3D and 2D virtual environments and interaction with object in a three dimensional virtual space accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the system that maps the inputs to the coordinates of the 3D UI element and the inputs to the coordinates of the 2D UI element by adjusting the dimensional values of the inputs as taught by Ofstad. After modification of WILLIAMS, the virtual cursor 60 that is presented in a 3D format can also be converted or adopted in a 2D format as taught by Ofstad. One would have been motivated to make such a combination in order to provide allowing the user to interact with objects at any distance within the virtual environment by improving the efficiency of a computer by allowing the use of a single pointing device to be used to interact with multiple virtualized displays and objects within a virtual environment.
WILLIAMS, Algreatly and Ofstad does not teach the method wherein:
tracking one or more of the three dimensional values that move the first flattened 2D shape away from the specific 2D plane at which the 2D UI element is presented in the 3D environment less than a threshold amount; and
changing the one or more of the three dimensional values issued from the input device to retain the first flattened 2D shape on the specific 2D plane;
attaching the pointer to the 3D UI element comprises presenting the pointer with a second volumetric shape having a proximal tip and a distal end;
However, ZHOU teaches the method comprising:
tracking one or more of the three dimensional values that move the first flattened 2D shape away from the specific 2D plane at which the 2D UI element is presented in the 3D environment less than a threshold amount (see ZHOU : Fig.6, [0082], “If it is determined that the final cursor ray R1 does not intersect with any selectable virtual object within the VR scene 175 (at step 620—No), the method 600 continues at step 630. At steps 630-645, the cursor position engine 113 performs an interpolation technique to interpolate the depth of the final cursor ray R1 based on the selectable virtual objects within the current VR scene 175.”) At step 630, the cursor position engine 113 projects the selectable objects of the current VR scene 175 and the [0083] cursor onto a 2D plane 500. The selectable objects of the VR scene 175 can be projected onto the 2D plane (x, y plane) based on 2D coordinates (x, y) derived from the 3D coordinates of the selectable objects. The cursor can be projected onto the 2D plane based on 2D (x, y) coordinates derived from the final cursor ray R1 321 determined for the current frame.”); and
changing the one or more of the three dimensional values issued from the input device to retain the first flattened 2D shape on the specific 2D plane (see ZHOU : Fig., [0083], “At step 630, the cursor position engine 113 projects the selectable objects of the current VR scene 175 and the [0083] cursor onto a 2D plane 500. The selectable objects of the VR scene 175 can be projected onto the 2D plane (x, y plane) based on 2D coordinates (x, y) derived from the 3D coordinates of the selectable objects. The cursor can be projected onto the 2D plane based on 2D (x, y) coordinates derived from the final cursor ray R1 321 determined for the current frame.”);
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the system that track and change the one or more of the three dimensional values issued from the input device to retain the first flattened 2D shape on the specific 2D plane as taught by ZHOU. One would have been motivated to make such a combination in order to provide user an effective techniques for integrating 2D input devices into 3D computing environments ( see Zhao:[0007])
Day teaches the method comprising:
attaching the pointer to the 3D UI element comprises presenting the pointer with a second volumetric shape having a proximal tip and a distal end (see Day: Fig.14A-14B, “illustrates an example of an environment including a cursor 1202 and an object 1204. In this example, the object 1204 is a cube sitting on a table. As the cursor 1202 transitions from position 1412 in FIG. 14A to position 1414 in FIG. 14B, the cursor 1202 moves over (or in front of) the object 1204. Accordingly, when the cursor 1202 is at position 1414, the cursor 1202 is blocking from the users view a portion of the object 1204, which as discussed above can be distracting to a user.”)
Because WILLIAMS, Algreatly, Ofstad, ZHOU and Day are in the same/similar field of endeavor of 3D and 2D virtual environments and interaction with object in a three dimensional virtual space accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the system that attach the pointer to the 3D UI element comprises presenting the pointer with a second volumetric shape having a proximal tip and a distal end as taught by Day. One would have been motivated to make such a combination in order to provide allowing the user to interact with objects at any distance within the virtual environment by improving the efficiency of a computer by allowing the use of a single pointing device to be used to interact with multiple virtualized displays and objects within a virtual environment.
Regarding Claim 4,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein changing the one or more of the three dimensional values (see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”), comprises:
receiving an input with a first coordinate value, a second coordinate value, and a third dimensional value for moving the pointer across the three dimensions of the 3D environments (see Algreatly: Fig.5.5 -5.3, [0044], “providing a movement along the positive x-axis, moves the pointer and the virtual camera along the positive x-axis of the hand-held device's display as illustrated in FIG. 5.2. Providing a movement along the negative x-axis, moves the pointer and the virtual camera along the negative x-axis of the hand-held device's display as illustrated in FIG. 5.3.”); and
converting at least one of the first coordinate value, the second coordinate value, and the third coordinate value to a value of the specific 2D plane about which the 2D UI element is positioned in the 3D environment (see Algreatly: Fig.4, [0078], “if the user pressed on the "+x" position of the 5-way button, which is the "east" direction of the 5-way digital button, then a (0,1,0,0,0) signal is generated, and if the user then pressed on the "+y" position of the 5-way button, which is the "north" direction, then a (1,0,0,0,0) signal is generated. Accordingly, the computer system translates these two positions pressing as a counter-clockwise rotation about the z-axis as described previously in the table of FIG. 4.”)
It would have been obvious to combine before the effective filing date of the invention because it provides provide users of virtual reality or augment reality application with an efficient input tool (cursor) to manipulate 2D or 3D user interface elements or objects using interactive cursor that is efficient, error free and precise cursor manipulation of virtual objects to enhanced and simplified the experience with the virtual/augmented reality environment.
Regarding Claim 5,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein changing the two or more of the different three dimensional values (see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”), comprises:
receiving an input that is defined across first, second, and third dimensions, wherein the input controls the movement of the pointer (see Algreatly: Fig.6, [0052], “if the endpoint of the pointer is intersected with the cube in node (0, 0, 0) and the pointer is rotated clockwise about the y-axis of the hand-held device's display, then the endpoint of the pointer will be moved parallel to the xy-plane of the cube, respectively, on nodes (1, 0, 0), (2, 0, 0), (3, 0, 0), (4, 0, 0), (5, 0, 0), (6, 0, 0), (6, 1, 0), (6, 2, 0), and (6, 3, 0). Also, if the pointer is rotated counter-clockwise about the x-axis of the hand-held device's display then the endpoint of the pointer will be moved on the yz-plane of the cube, respectively, on nodes (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 1, 6), (0, 2, 6), and (0, 3, 6).”);
determining that a value defined for the first coordinate of the input matches a first coordinate value of a particular 3D coordinate from the coordinates of the 3D UI element (see Algreatly: Fig.6, [0053], “Each spot in the 3D virtual environment that may be targeted to interact with the pointer needs at least one node to be located inside it, where, this is it to enable reaching these spots when the pointer is rotated or moved in 3D on the hand-held device's display. Accordingly, it is possible, in some cases, to reduce the number of nodes to a minimum number that is equal to the number of the targeted spots.”); and
adjusting values that are defined for the second dimension and the third dimension of the input to match a second coordinate value and a third coordinate value of the particular 3D coordinate (see Algreatly: Fig.6, [0052], “Also, if the pointer is rotated counter-clockwise about the x-axis of the hand-held device's display then the endpoint of the pointer will be moved on the yz-plane of the cube, respectively, on nodes (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 1, 6), (0, 2, 6), and (0, 3, 6).”);”
See Claim 1 above the motivation to combine WILLIAMS, and Algreatly.
Regarding Claim 6,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein:
moving first flattened 2D shape comprises moving the pointer according to a first coordinate value and a second coordinate value of the first inputs while adjusting a third coordinate value of the first inputs to retain the pointer on the specific 2D surface of the 2D UI element (see Algreatly: Fig.5.1-5.3, [0044], “providing a movement along the positive x-axis, moves the pointer and the virtual camera along the positive x-axis of the hand-held device's display as illustrated in FIG. 5.2. Providing a movement along the negative x-axis, moves the pointer and the virtual camera along the negative x-axis of the hand-held device's display as illustrated in FIG. 5.3.”); and
wherein moving the second volumetric shape comprises moving the pointer according to a third dimensional value of the second inputs while adjusting a second dimensional value and the first dimensional value of the second inputs to retain the pointer on the 3D surface of the 3D UI element (see Algreatly: Fig.6, [0052], “Providing a movement along the positive z-axis, moves the virtual camera forward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.6. Provide a movement along the negative z-axis, moves the virtual camera backward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.7.”)
One would have been motivated to combine WILLIAMS, Algreatly, Ofstad and Day, before the effective filing date of the invention because it provides provide users of virtual reality or augment reality application with an efficient input tool (cursor) to manipulate 2D or 3D user interface elements or objects using interactive cursor that is efficient, error free and precise cursor manipulation of virtual objects to enhanced and simplified the experience with the virtual/augmented reality environment.
Regarding Claim 7,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein:
changing the one or more of the three dimensional values (see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”), comprises:
determining a first coordinate value and a second coordinate value from the inputs that match to a first dimension and a second dimension defined for a particular 2D coordinate of the 2D UI element and a third coordinate value from the first inputs that does not match a third dimension at which the particular 2D coordinate is presented in the 3D environment (see Algreatly: Fig.5.6-5.7, [0042], “Providing a movement along the positive z-axis, moves the virtual camera forward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.6. Provide a movement along the negative z-axis, moves the virtual camera backward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.7.”); and
adjusting the third coordinate value to a value for the third dimension at which the particular 2D coordinate is presented in the 3D environment (see Algreatly: Fig.5.6 -5.7, [0042], Providing a clockwise rotation about the x-axis, rotates the pointer and the virtual camera clockwise about the x-axis of the hand-held device's display as illustrated in FIG. 5.8. Providing a counter-clockwise rotation about the x-axis, rotates the pointer and the virtual camera counter-clockwise about the x-axis of the hand-held device's display as illustrated in FIG. 5.9.”)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the teaching of Algreatly because it provides provide users of virtual reality or augment reality application with an efficient input tool (cursor) to manipulate 2D or 3D user interface elements or objects using interactive cursor that is efficient, error free and precise cursor manipulation of virtual objects to enhanced and simplified the experience with the virtual/augmented reality environment. (See Algreatly [0006])
Regarding Claim 8,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 7. Algreatly further teaches the method wherein:
hanging the two or more of the different three dimensional values (see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”), comprises:
determining that a third dimensional value from second the inputs matches with a first dimension defined for a particular 3D coordinate of the 3D UI element and the second dimensional value and a first dimensional value of the second inputs that do not match with second and third dimensions defined for the particular 3D coordinate (see Algreatly: Fig.6, [0052], For example, if the endpoint of the pointer is intersected with the cube in node (0, 0, 0) and the pointer is rotated clockwise about the y-axis of the hand-held device's display, then the endpoint of the pointer will be moved parallel to the xy-plane of the cube, respectively, on nodes (1, 0, 0), (2, 0, 0), (3, 0, 0), (4, 0, 0), (5, 0, 0), (6, 0, 0), (6, 1, 0), (6, 2, 0), and (6, 3, 0). ; and
adjusting a second dimensional value and the third dimensional value of the second input to values for the second dimension and the third dimension of the particular 3D coordinate (see Algreatly: Fig.6, [0052], “, if the pointer is rotated counter-clockwise about the x-axis of the hand-held device's display then the endpoint of the pointer will be moved on the yz-plane of the cube, respectively, on nodes (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 1, 6), (0, 2, 6), and (0, 3, 6).”)
See Claim 1 above the motivation to combine WILLIAMS, Algreatly, Ofstad ZHOU and Day.
Regarding Claim 9,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein:
mapping the inputs to the coordinates of the 2D UI element (see Algreatly: Fig.3, [0042], “operating said 5-way button requires the "+x", "-x", "+y", and "-y" positions to have elevated level than the "z" position. This is to achieve two goals: the first goal is to avoid hitting the "z" position by mistake while moving the user's finger from the "+x" to the "-x" position or vice versa, or from the "+y" to "-y" position or vice versa. The second goal is to make the user distinguish the difference between moving in the y-axis or the z-axis, where the height of the "z" position is lower than the "y" position and the "-y" position. However, most of the 5-way buttons that are included on the hand-held device's keyboard have such dual-level configuration.”), comprises:
positioning the pointer on a particular 2D coordinate from the coordinates of the 2D UI element that is presented at a particular 3D coordinate in the 3D environment based on the pointer being attached to the 2D UI element, the first inputs comprising a first coordinate value and a second dimensional value that match first and second dimensions of the particular 3D coordinate and a third dimensional value that does not match a third dimension of the particular 3D coordinate (see Algreatly: Fig.5.6-5.7, [0042], “Providing a movement along the positive z-axis, moves the virtual camera forward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.6. Provide a movement along the negative z-axis, moves the virtual camera backward, parallel to the direction of the pointer in 3D on the hand-held device's display as illustrated in FIG. 5.7.”), and said adjusting of the particular dimensional value that modifies the third coordinate value to match the third dimension of the particular 3D coordinate
(see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the teaching of Algreatly because it provides provide users of virtual reality or augment reality application with an efficient input tool (cursor) to manipulate 2D or 3D user interface elements or objects using interactive cursor that is efficient, error free and precise cursor manipulation of virtual objects to enhanced and simplified the experience with the virtual/augmented reality environment. (See Algreatly [0006])
Regarding Claim 10,
WILLIAMS, Algreatly, Ofstad , ZHOU and Day teaches all the limitations of Claim 9. Algreatly further teaches the method wherein:
mapping the inputs to the coordinates of the 3D UI element (see Algreatly: Fig.6, [0051], “illustrates a cube divided by a plurality of intersected hidden lines parallel to the x, y, and z-axis to form a number of nodes 160. As shown in the figure; the cube indicates numerals that represent the coordinates of the x, y, and z-axis. The mesh grid enables the endpoint of the pointer to target any spot in the virtual 3D environment on the hand-held device's display without any complex mathematical calculations.”), further comprises:
positioning the pointer on a specific 3D coordinate from the coordinates of the 3D UI element based on the pointer being attached to the 3D UI element, the second inputs comprising a third dimensional value that matches a first dimension of the specific 3D coordinate and a second dimensional value and a first dimensional value that do not match a second dimension and a third dimension of the specific 3D coordinate, (see Algreatly: Fig.5.6-5.7, [0052], “For example, if the endpoint of the pointer is intersected with the cube in node (0, 0, 0) and the pointer is rotated clockwise about the y-axis of the hand-held device's display, then the endpoint of the pointer will be moved parallel to the xy-plane of the cube, respectively, on nodes (1, 0, 0), (2, 0, 0), (3, 0, 0), (4, 0, 0), (5, 0, 0), (6, 0, 0), (6, 1, 0), (6, 2, 0), and (6, 3, 0). Also, if the pointer is rotated counter-clockwise about the x-axis of the hand-held device's display then the endpoint of the pointer will be moved on the yz-plane of the cube, respectively, on nodes (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 1, 6), (0, 2, 6), and (0, 3, 6).”) and said adjusting of the first and second dimensional values to match the second dimension and the third dimension of the specific 3D coordinate (see Algreatly: Fig.3, [0040], “illustrates a table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent moving along the x, y, or z-axis. FIG. 4 illustrates another table that indicates the user's finger movement or pressing on the five positions of the 5-way button to represent rotating about the x, y, or z-axis. As shown in these two tables each degree of freedom is provided by one alternative of the user's finger movement or pressing, except rotating about the z-axis which can be provided by four different alternatives of the user finger's movements.”).
One would have been motivated to combine WILLIAMS, Algreatly, Ofstad , ZHOU and Day, before the effective filing date of the invention because it provides provide users of virtual reality or augment reality application with an efficient input tool (cursor) to manipulate 2D or 3D user interface elements or objects using interactive cursor that is efficient, error free and precise cursor manipulation of virtual objects to enhanced and simplified the experience with the virtual/augmented reality environment.
Regarding Claim 12,
As shown above WILLIAMS, Algreatly, Ofstad , ZHOU and Day teaches all the limitations of claim 1. Ofstad further teaches the method further comprising:
changing the pointer from a default shape to a first flattened 2D shape in response to attaching the pointer to the 2D UI element; and changing the pointer from the default shape to the volumetric shape in response to attaching the pointer to the 3D UI element (see Ofstad: Fig.1A, [0020], “A processor, such as processor 225 of FIG. 1A, may render a 3D scene and 3D geometric shapes (i.e., a 3D cursor and 3D object) within the 3D scene on a 2D display using an isometric projection. In another implementation, the processor 225 may render the 3D geometric shapes on a 2D display using a two-point perspective. More generally, the processor 225 may render a display using any desired level of detail to illustrate a 3D cursor and objects for which 3D geometry data is available. The isometric projections, two-point perspectives and any other rendering may be based on either (x, y) coordinate data or (x, y, z) coordinate data.”)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the method that provides an adaptive cursor that based on the interacted user interface elements as taught by Ofstad. One would have been motivated to make such a combination in order to provide users an efficient, error free and precise cursor manipulation of virtual objects to enhance the experience with the virtual/augmented reality environment.
Regarding independent Claims 14 and 20,
Claim 14 is directed to a system claim and Claim 20 is directed to A non-transitory computer-readable medium and the claims have similar/same claim limitation as Claim 1 and are rejected with the same rationale.
Regarding Claim 17-19,
Claim 17-19 are directed to a system claim the claims have similar/same claim limitation as Claim 4-6 and are rejected with the same rationale.
Regarding Claim 21,
WILLIAMS, Algreatly, Ofstad , ZHOU and Day teaches all the limitations of Claim 1. Algreatly further teaches the method wherein:
determining that the pointer is within a threshold distance of the 2D UI element and the 3D UI element and that the pointer is closer to the 3D UI element than the 2D UI element (see Day: Fig.51, [0747], “moving direction and the moving distance in the depth direction (z direction) are calculated (determined) based on a rotation direction and a rotation angle of a wheel 202A of the mouse 202 when rotating the wheel 202A while pushing a predetermined key 201A such as a control key (Ctrl key) of the keyboard 201.”) and
changing a color of the pointer and a border of the 3D UI element to a matching color that is different than a color of a border of the 2D UI element (see Day: Fig.54, [0743], “shows a front view and a right-side view showing change in the three-dimensional space when pointing at an object in front of the pointer.”)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the system that determining that the pointer is within a threshold distance of the 2D UI element and the 3D UI element and that the pointer is closer to the 3D UI element than the 2D UI element and changing a color of the pointer and a border of the 3D UI element as taught by Day. One would have been motivated to make such a combination in order to provide allowing the user to interact with objects at any distance within the virtual environment by improving the efficiency of a computer by allowing the use of a single pointing device to be used to interact with multiple virtualized displays and objects within a virtual environment.
Regarding Claim 22,
WILLIAMS, Algreatly, Ofstad and Day teaches all the limitations of Claim 7. Algreatly further teaches the method wherein:
determining that the pointer partially overlaps with the 2D UI element and the 3D UI element and that the pointer overlaps with more of the 3D UI element than the 2D UI element (see Day: Fig.53, [0754], “the pointer 4 that overlaps on the object 5 points at the object 5.”] and
presenting the pointer and the 3D UI element with a matching color that is different than a color with which the 2D UI element is presented (see Day: Fig.53, [0754], “by changing the color of the object 5 when the pointer 4 points at the object 5, it can be recognized intuitively and accurately whether the pointer 4 that overlaps on the object 5 points at the object 5.”)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of WILLIAMS to include the system that determining that the pointer partially overlaps with the 2D UI element and the 3D UI element and presenting the pointer and the 3D UI element with a matching color that is different than a color with which the 2D UI element is presented as taught by Day. One would have been motivated to make such a combination in order to provide allowing the user to interact with objects at any distance within the virtual environment by improving the efficiency of a computer by allowing the use of a single pointing device to be used to interact with multiple virtualized displays and objects within a virtual environment.
Regarding Claim 23,
WILLIAMS, Algreatly, Ofstad, ZHOU and Day teaches all the limitations of Claim 7. Algreatly further teaches the method wherein:
changing the pointer to have a flat shape with the proximal tip and the distal end in response to attaching the pointer to the 2D UI element, wherein the flat shape entirely contacts the 2D surface of the 2D UI element (see Williams: Fig.9A-9C, [0129], “the user may operate the user input device to move the cursor 910 so that the cursor 910 is pointing at 2D element 906 (which is an image of a mobile phone in the illustrated example). In response to the cursor 910 being in an associated position with respect to the 2D element 906, the processing unit 130 then provides an indicator 930 (e.g., generated by the graphic generator 430) for display by the screen 902.”); and
changing the pointer to have a 3D shape with the selection tip in response to attaching the pointer to the 3D UI element, wherein the proximal tip of the 3D shape contacts the 3D surface of the 3D UI element and a remainder of the 3D shape floats above and does not contact the 3D surface of the 3D UI element (see WILLIAMS: Fig.9A-9C, [0131], “the user may position the object 950 (e.g., in one or more directions) and/or rotate the object 950 (e.g., about one or more axes) based on the 3D model. For example, as shown in FIGS. 9C-9D, the user may use the cursor 910 to select the object 950 and move the object 950 to a certain location with respect to the environment as viewed through the screen 902. The user may also operate the user input device to define an axis of rotation to rotate the object 950. In response to a signal (received from the user input device) that prescribes rotation of the object 950, the graphic generator renders the object 950 base