Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 4/6/26 has been entered and made of record. Claims 1-2, 4-6, 9-15 and 17-18 are amended. Claim 20 is new. Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments with respect to claims 1 and 18 have been considered but they are not persuasive.
Applicant asserts that independent claims 1 and 18 to recite that the electronic device is configured to receive, as one or more touch gestures, "control of a three-dimensional (3D) virtual reality space and/or an object sketched within the 3D virtual reality space." Tosas does not disclose a configuration in which a user sketches objects within a 3D virtual reality space and controls those sketched objects. Bedikian does not supply the features missing from Tosas (p. 7-10 of Remarks).
Examiner notices that Kiemele et al. (US 2019/0065026 A1) discloses “Input device 110 includes a touch sensor configured to receive user input, which may be provided to HMD device 104 and used to control the 3D virtual reality experience by varying visual content 106 rendered on the HMD display… Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions” in [0015]; “For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402, which is rendered in region 108 on HMD device 104…As shown, HMD device 104 may render an icon 412 in region 108 representing the selected stroke/pattern at 3D location corresponding to the last location where input was applied in canvas portion 402.” in [0027]; see also [0032]. Here, Kiemele clearly teaches user can sketch objects with a 3D virtual reality space.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tosas Bautista (US 2014/0248950 A1) in view of Kiemele et al. (US 2019/0065026 A1).
As to Claim 1, Tosas Bautista teaches An electronic system comprising:
an electronic device configured to receive, as one or more touch gestures, control of a virtual reality space and/or an object within the virtual reality space, and display a first screen according to a corresponding first view frustum within the 3D virtual reality space (Tosas Bautista discloses “If the mobile device's keypad is a touch screen, the user will be able to use traditional navigation gestures to zoom in and zoom out (pinch gesture), roll (rotation gesture), pan (pan gesture) and click (tap gesture)” in [0097]; “Embodiments of the invention estimate the pose of the mobile device within the defined world coordinate system and render a perspective view of the visual output mapped on the virtual surface onto the mobile device's display according to the estimated pose” in [0387]; see also Fig 2B); and
a head-mounted display (HMD) configured to be worn on a head of a user and to display a second screen of the 3D virtual reality space according to a direction of the head of the user from a viewpoint corresponding to a second view frustum within the 3D virtual reality space, wherein a virtual display of the electronic device displayed on the second screen according to the direction of the head is synchronized to an actual display of the electronic device (Tosas Bautista discloses “FIG. 4I depicts a typical configuration of an embodiment of the system showing the relative positions of a user 200, HMD 450, mobile device 100, and a representation of the virtual surface 201. The pose 45I of the HMD, the pose 203 of the mobile device, and the representation of the virtual surface 201 can all be defined within a common world coordinate system 202” in [0111], 3D mesh in [0118], see also Fig 4I as below:
PNG
media_image1.png
333
448
media_image1.png
Greyscale
), and
when the user performs a motion of inputting one or more touch gestures into the virtual display displayed on the second screen, the one or more touch gestures are input into the actual display of the electronic device so that control of the virtual reality space and/or the object is performed (Tosas Bautista discloses “For example, if the mobile device has a touchscreen, the user can perform navigation gestures such as sliding (for scrolling), pinching (for zooming) and rotating in order to visualise the information more clearly” in [0077]; “If the mobile device is used as display, the perspective projection of the virtual surface will be shown on the mobile device's display according to the pose 203 of the mobile device within the common world coordinate system 202. In this scenario, the HMD can show extra information relating to the contents mapped on the virtual surface. For example, the HMD can show a high level map of all the contents on the virtual surface, indicating the region that is currently being observed on the mobile device's display” in [0112].)
Tosas Bautista doesn’t explicitly teach control of a three-dimensional (3D) virtual reality space and/or an object sketched within the 3D virtual reality space, the one or more touch gestures are input into the actual display of the electronic device at a physical location on the actual display that corresponds to the spatial location on the virtual display so that control of the 3D virtual reality space and/or the object is performed. Kiemele further discloses “User input received by the touch sensor may include touch, hover, and/or hand gesture input, for example, and may be sensed via any suitable touch sensing technology… Input device 110 includes a touch sensor configured to receive user input, which may be provided to HMD device 104 and used to control the 3D virtual reality experience by varying visual content 106 rendered on the HMD display… Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions” in [0015]; “HMD device 104 may track touch-sensitive input device 110 and render a user interface 112 based on the tracking of the input device… HMD device 104 may render any suitable user interface and/or representation of input device 110, further examples of which are described below. Other representations may be rendered as well, such as a representation of the hands of user 102.” in [0016]; “For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402, which is rendered in region 108 on HMD device 104…As shown, HMD device 104 may render an icon 412 in region 108 representing the selected stroke/pattern at 3D location corresponding to the last location where input was applied in canvas portion 402.” in [0027]; “When user input received at touch-sensitive input device 110 is supplied to a user interface rendered in a virtual reality experience, HMD device 104 may perform a mapping step in supplying the user input to the user interface. This may include mapping a location of the user input on input device 110 to a location in the user interface, which may utilize any suitable mapping function(s).” in [0034], see also [0032]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tosas Bautista with the teaching of Kiemele so as to provide a user interface to facilitate the generation of new or editing of visual content sketched by hand gesture.
As to Claim 2, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the virtual display of the electronic device displayed on the second screen continuously displays the first screen displayed on the actual display of the electronic device in real-time such that changes to the first screen are immediately reflected in the virtual display (Tosas Bautista discloses “the perspective projection of the virtual surface will be shown on the mobile device's display according to the pose 203 of the mobile device within the common world coordinate system 202. In this scenario, the HMD can show extra information relating to the contents mapped on the virtual surface. For example, the HMD can show a high level map of all the contents on the virtual surface, indicating the region that is currently being observed on the mobile device's display” in [0112]. Kiemele, Fig 4.)
As to Claim 3, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein one or a combination of at least two of a size, a position, and an orientation of the virtual display of the electronic device displayed on the second screen is synchronized with that of the actual display of the electronic device (Tosas Bautista discloses “For example, the pose 203 of the mobile device, estimated within the common world coordinate system 202, can be used to control the level of zoom of the part of the virtual surface shown on the HMD, while the pose 451 of the HMD, estimated within the common world coordinate system 202, can be used to control the rotation of the part of the virtual surface shown on the HMD” in [0110]; “the perspective projection of the virtual surface will be shown on the mobile device's display according to the pose 203 of the mobile device within the common world coordinate system 202. In this scenario, the HMD can show extra information relating to the contents mapped on the virtual surface. For example, the HMD can show a high level map of all the contents on the virtual surface, indicating the region that is currently being observed on the mobile device's display.” in [0112].)
As to Claim 4, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the hand of the user being tracked is displayed together with the virtual display of the electronic device on the second screen, and a touch gesture is input into the actual display with a motion of the user inputting the touch gesture into the virtual display (Tosas Bautista discloses “These, also known as "natural interfaces", may involve tracking of the users hands and body allowing them to directly "touch" the information or objects overlaid on their fields of view.” in [0005]. Kiemele also discloses “HMD device 104 may render any suitable user interface and/or representation of input device 110, further examples of which are described below. Other representations may be rendered as well, such as a representation of the hands of user 102” in [0016].)
As to Claim 5, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to perform moving and/or rotating the first view frustum within the 3D virtual reality space based on the one or more touch gestures input from the user, and the moving and/or rotating of the first view frustum is performed within a preset boundary within the 3D virtual reality space (Tosas Bautista discloses “If the mobile device's keypad is a touch screen, the user will be able to use traditional navigation gestures to zoom in and zoom out (pinch gesture), roll (rotation gesture), pan (pan gesture) and click (tap gesture). Pitch and yaw rotation could be achieved by first setting a separate mode, and then using the pan gesture instead” in [0097], see also Fig 2 & 4. Kiemele, Fig 4.)
As to Claim 6, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to perform a sketch on a screen plane of the first view frustum within the 3D virtual reality space based on a pen drawing input into the actual display, and display the sketched screen plane as the first screen (Kiemele discloses “User interface 400 includes one or more controls for rendering new visual content. For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402, which is rendered in region 108 on HMD device 104” in [0027], see also Fig 4 below:
PNG
media_image2.png
732
589
media_image2.png
Greyscale
).
As to Claim 7, Tosas Bautista in view of Kiemele teaches The electronic system of claim 6, wherein the electronic device is configured to perform one or a combination of at least two of moving, rotating, and resizing the object on the screen plane based on one or more touch gestures on an object generated by the sketch (Tosas Bautista discloses finger gestures such as pinching and rotating in [0003]. Kiemele further discloses “A touch-sensitive input device 110 may facilitate interaction with the virtual reality experience. Input device 110 includes a touch sensor configured to receive user input, which may be provided to HMD device 104 and used to control the 3D virtual reality experience by varying visual content 106 rendered on the HMD display. User input received by the touch sensor may include touch, hover, and/or hand gesture input…” in [0015]; translational and rotational input in [0011].)
As to Claim 8, Tosas Bautista in view of Kiemele teaches The electronic system of claim 6, wherein the electronic device is configured to determine a projection plane of an object generated by the sketch based on a viewing direction of the first view frustum when performing the sketch (Kiemele discloses “User interface 400 includes one or more controls for rendering new visual content. For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402, which is rendered in region 108 on HMD device 104” in [0027]; translational and rotational input in [0011].)
As to Claim 9, Tosas Bautista teaches in view of Kiemele The electronic system of claim 8, wherein the electronic device is configured to generate the object on the projection plane and place the generated object in the 3D virtual reality space when a command to generate the object is input from the user (Kiemele discloses “As described above, shape 404 may represent a 2D or 3D input applied to canvas portion 402. Various approaches may enable the reception of 3D input at canvas portion 402. When touch-sensitive input device 110 is operable to detect a z-coordinate of touch input into/out of the input device---e.g., as a force or pressure-the user may control such force/pressure to vary the z-coordinate of input applied to canvas portion 402.” in [0029].)
As to Claim 10, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to, based on one or more touch gestures on a card comprising an object within the 3D virtual reality space, perform one or a combination of at least two of moving, rotating, and resizing the object on a first reference plane in which the card is included (Tosas Bautista discloses “If the mobile device's keypad is a touch screen, the user will be able to use traditional navigation gestures to zoom in and zoom out (pinch gesture), roll (rotation gesture), pan (pan gesture) and click (tap gesture)” in [0097]; see also [0003-0004, 0077].)
As to Claim 14, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein one or a combination of at least two of moving, rotating, and resizing the object according to one or more touch gestures input from the user is performed within a preset boundary within the3D virtual reality space (Tosas Bautista discloses “If the mobile device's keypad is a touch screen, the user will be able to use traditional navigation gestures to zoom in and zoom out (pinch gesture), roll (rotation gesture), pan (pan gesture) and click (tap gesture)” in [0097].)
As to Claim 15, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the first screen and/or the second screen displays a third view frustum corresponding to a second electronic device accessing the 3D virtual reality space in a viewing direction of the third view frustum at a position corresponding to the second electronic device (Tosas Bautista discloses “FIG. 4I depicts a typical configuration of an embodiment of the system showing the relative positions of a user 200, HMD 450, mobile device 100, and a representation of the virtual surface 201. The pose 45I of the HMD, the pose 203 of the mobile device, and the representation of the virtual surface 201 can all be defined within a common world coordinate system 202” in [0111]. Here, the pose 451 can have one or more view frustum.)
As to Claim 16, Tosas Bautista in view of Kiemele teaches The electronic system of claim 15, wherein the first screen and/or the second screen, in response to a sketch being performed on a second screen plane of the third view frustum based on a pen drawing input into an actual display of the second electronic device, displays the sketched second screen plane (Tosas Bautista discloses “If the mobile device is used as display, the perspective projection of the virtual surface will be shown on the mobile device's display according to the pose 203 of the mobile device within the common world coordinate system 202. In this scenario, the HMD can show extra information relating to the contents mapped on the virtual surface. For example, the HMD can show a high level map of all the contents on the virtual surface, indicating the region that is currently being observed on the mobile device's display” in [0112]. Kiemele further discloses “User interface 400 includes one or more controls for rendering new visual content. For example, user interface 400 may include a canvas portion 402 in which a user can draw 2D or 3D shapes and images via the application of touch input. FIG. 4 shows one such possible shape 404 drawn in canvas portion 402, which is rendered in region 108 on HMD device 104” in [0027], see also Fig 4.)
As to Claim 17, Tosas Bautista in view of Kiemele teaches The electronic system of claim 15, wherein the electronic device is configured to, based on one or more pinch gestures on an object placed within the virtual reality space, input from the user wearing the HMD, perform one or a combination of at least two of moving, rotating, and resizing the object (Tosas Bautista discloses “If the mobile device's keypad is a touch screen, the user will be able to use traditional navigation gestures to zoom in and zoom out (pinch gesture), roll (rotation gesture), pan (pan gesture) and click (tap gesture)” in [0097].)
Claim 18 recites similar limitations as claim 1 but in a method form. Therefore, the same rationale used for claim 1 is applied.
Claim 19 recites similar limitations as claim 1 but in a computer readable medium form. Therefore, the same rationale used for claim 1 is applied.
As to Claim 20, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the virtual display is synchronized with the actual display of the electronic device by: (i) displaying in the virtual display the same visual content currently shown on the actual display, and (ii) establishing a spatial-to-physical mapping between spatial positions within the virtual display and corresponding physical positions on the actual display, and wherein when the user performs the motion of touching with the hand the spatial location on the virtual display, the physical location on the actual display is determined based on the spatial-to-physical mapping (Kiemele discloses “When user input received at touch-sensitive input device 110 is supplied to a user interface rendered in a virtual reality experience, HMD device 104 may perform a mapping step in supplying the user input to the user interface. This may include mapping a location of the user input on input device 110 to a location in the user interface, which may utilize any suitable mapping function(s). As examples, functions may be used that apply acceleration or otherwise alter kinematic properties associated with the user input, scale a magnitude of the user input, and/or apply user interface magnetism that preferentially biases the mapping toward a location.” in [0034]; “In such an example, the instructions alternatively or additionally may be executable to map a location of the user input on the touch-sensitive input device to a location in the user interface. In such an example, the 3D location in which the user interface is rendered alternatively or additionally may correspond to a real-world location of a display of the touch-sensitive input device.” in [0074].)
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Tosas Bautista in view of Kiemele and Bedikian et al. (US 2014/0201666 A1).
As to Claim 11, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to, based on one or more touch gestures on a stand of a card comprising an object within the 3D virtual reality space, perform moving and/or rotating the object on a second reference plane in which the stand is included, and the stand is generated to be parallel to a bottom of the 3D virtual reality space at a position at which the card is projected onto the bottom of the 3D virtual reality space (Tosas Bautista discloses “For example, if the mobile device has a touchscreen, the user can perform navigation gestures such as sliding (for scrolling), pinching (for zooming) and rotating in order to visualise the information more clearly. Pinching and rotating are examples of navigation gestures that require the use of two fingers in order to be performed. Two finger navigation gestures usually require the involvement of both hands, one to hold the mobile device and another to perform the gesture” in [0077]; “preferred embodiments of the system use an expanding plane 204, located at the origin of the world coordinate system. In these embodiments, the Photomap image can be thought of as a patch of texture anchored on this plane” in [0323]. Here, Tosas Bautista teaches a virtual object is anchored on a real plane. It is obvious that the virtual object can be a virtual image frame or card anchored on a real surface. For example, Bedikian discloses “In embodiments, the virtual surface construct may be fixed in space, e.g., relative to the screen” in [0105]; “As another example, the user's hand may be tracked to determine the positions and orientations of all fingers; each finger may have its own associated virtual surface construct… A joint virtual plane may serve, e.g., as a virtual drawing canvas on which multiple lines can be drawn by the fingers at once” in [0127], see also [0070].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tosas Bautista and Kiemele with the teaching of Bedikian so as to provide a gesture control on a drawing canvas.
As to Claim 12, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to, while a first touch gesture on a stand of a card comprising an object within the 3D virtual reality space is input, control a height of the card within the 3D virtual reality space based on a second touch gesture on the card (Tosas Bautista discloses “For example, if the mobile device has a touchscreen, the user can perform navigation gestures such as sliding (for scrolling), pinching (for zooming) and rotating in order to visualise the information more clearly. Pinching and rotating are examples of navigation gestures that require the use of two fingers in order to be performed. Two finger navigation gestures usually require the involvement of both hands, one to hold the mobile device and another to perform the gesture” in [0077]. Bedikian further discloses “In an embodiment, once engaged, further movements of the control object may serve to move graphical components across the screen (e.g., drag an icon, shift a scroll bar, etc.), change perceived "depth" of the object to the viewer (e.g., resize and/or change shape of objects displayed on the screen in connection, alone, or coupled with other visual effects) to create perception of "pulling" objects into the foreground of the display or "pushing" objects into the background of the display, create new screen content (e.g., draw a line)” in [0106]; “Two parallel virtual planes may also be used to, effectively, define a virtual control construct with a certain associated thickness (i.e., a "virtual slab")” in [0107]; “To give just one example, the user may, with her index finger stretched out, have her thumb and middle finger touch so as to pin the virtual surface construct at a certain location relative to the current position of the index-finger-tip” in [0111].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tosas Bautista and Kiemele with the teaching of Bedikian so as to enable the user to intuitively control and manipulate the electronic device and virtual objects by simply performing body gestures (Bedikian, [0150]).
As to Claim 13, Tosas Bautista in view of Kiemele teaches The electronic system of claim 1, wherein the electronic device is configured to, while a first touch gesture on a stand of a card comprising an object within the 3D virtual reality space is input, perform rotating the object based on a second touch gesture on an area other than the card or the stand (Tosas Bautista discloses “For example, if the mobile device has a touchscreen, the user can perform navigation gestures such as sliding (for scrolling), pinching (for zooming) and rotating in order to visualise the information more clearly. Pinching and rotating are examples of navigation gestures that require the use of two fingers in order to be performed.” in [0077]. Bedikian further discloses “A joint virtual plane may serve, e.g., as a virtual drawing canvas on which multiple lines can be drawn by the fingers at once” in [0127]; gesture-recognition module to recognize rotation and translation gesture in [0070]. Official notice has been taken of the fact that a rotation gesture is performed by one finger on the object and another finger on the area other than the object, which is well-known in the art (see MPEP 2144.03).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tosas Bautista and Kiemele with the teaching of Bedikian so as to provide a rotation gesture control on a drawing canvas.
Conclusion
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Weiming He/
Primary Examiner, Art Unit 2611
Prosecution Insights