DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/16/2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-8, 17, 19-20, and 22-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2011/0205242 to Friesen et al. (Friesen), US 2018/0239515 A1 to Cooper et al. (Cooper), US 2017/0315692 to Wareing et al. (Wareing), and US 2020/0038120 A1 to Ziraknejad et al. (Ziraknejad).
As to claim 1, Friesen discloses a method comprising:
at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices:
displaying, via the display device (104, 240), a first plurality of graphical elements (302, 306, 308, 320, 322, 324) associated with a first plurality of output modalities associated with a manner in which a physical object (108) interacts with an extended reality (XR) environment (240) (Figs. 1-6, Pars. 66-69, see also Figs. 7-18);
while displaying the first plurality of graphical elements in the extended reality (XR) environment (240), detecting first movement of the physical object (108) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69); and
in response to detecting the first movement of the physical object (108):
in accordance with a determination that the first movement of the physical object causes the physical object (108) to breach a three-dimensional activation region (region within 302, 306, 308, 320, 322, 324) for a first graphical element among the first plurality of graphical elements (302, 306, 308, 320, 322, 324) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69), selecting a first output modality (e.g. line thickness, color, predesigned graphics, etc.) associated with the first graphical element as a current output modality for the physical object (108)(Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69);
in accordance with a determination that the first movement of the physical object causes the physical object to breach a three-dimensional activation region for a second graphical element among the first plurality of graphical elements (302, 306, 308, 320, 322, 324) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69), selecting a second output modality (e.g. line thickness, color, predesigned graphics, etc.) associated with the second graphical element as the current output modality for the physical object. (108) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69);
Friesen does not expressly disclose wherein the extended reality (XR) environment includes a representation of the physical object;
maintaining display of the first graphical element adjacent to the representation of the physical object within the extended reality (XR) environment; and
ceasing display of a remainder of the first plurality of graphical elements that does not include the first graphical element; and
maintaining display of the second graphical element adjacent to the representation of the physical object within the extended reality (XR) environment; and
ceasing display of a remainder of the first plurality of graphical elements that does not include the second graphical element; and
a three-dimensional activation region.
Cooper discloses wherein the extended reality (XR) environment (e.g. 500, 600, 800, 900, 1000, etc.) includes a representation of the physical object with a tip (e.g. brush 852 with tip 854) (Figs. 9-13, Pars. 54, 56, e.g. paint color 1152 would be applied/overlaid at the current location of the brush tip 854; see also Pars. 42-43);
maintaining display of the first graphical element (color 1152) overlaid on the tip of the representation of the physical object (854) within the extended reality (XR) environment (e.g. 500, 600, 800, 900, 1000, etc.)(Figs. 9-13, Pars. 54, 56; see also Pars. 42-43, 51); and
maintaining display of the second graphical element (852) overlaid on the tip of the representation of the physical object (e.g. 502, 804) within the extended reality (XR) environment (e.g. 500, 600, 800, 900, 1000, etc.)(Figs. 9-13, Pars. 54-56; see also Pars. 42-43, 51).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Friesen with the teaching of Cooper to provide more intuitive control for users to interact with computer-generated environments as suggested by Cooper (Par. 4).
Wareing discloses maintaining display of the first graphical element (e.g. 610 or 650) adjacent to the physical object (e.g. finger drag out the selected tool) (Figs. 6-7, 10, Pars. 73-75, 85); and ceasing display of a remainder of the first plurality of graphical elements (612, 614, 616, 618, 620) that does not include the first graphical element (610, 650) (Figs. 6-7, Pars. 73-75); and ceasing display of a remainder of the first plurality of graphical elements (e.g. 610, 614, 616, 618, 620) that does not include the second graphical element (e.g. 612) (Figs. 6-7, Pars. 73-75).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Friesen with the teaching of Wareing to provide more display working space to a user by hiding other non-used tools (Par. 74).
Ziraknejad discloses a three-dimensional activation region (e.g. three-dimensional control menu with selectable menu item in a volumetric spatial region) (Fig. 1B, abstract; Pars. 7-8, 38, 41).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Friesen with the teaching of Ziraknejad to provide an improved touchless input as suggested by Ziraknejad (Par. 6).
As to claims 17 and 20, see claim 1 rejection and motivation above.
As to claim 2, Friesen discloses the first and second output modalities cause different visual changes within the XR environment (e.g. line thickness, color, predesigned graphics, etc.) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69).
As to claims 19 and 22, see claim 7 rejection above.
As to claim 4, Friesen as modified discloses after selecting the first output modality associated with the first graphical element as the current output modality for the physical object, detecting second movement of the physical object (Figs. 5-6, 8-13, Pars. 42-43, 50, 52), Wareing’s Figs. 6-7, Pars. 73-75); and in response to detecting second movement of the physical object, moving the first graphical element based on the second movement of the physical object in order to maintain display of the first graphical element overlaid on the tip of the representation of the physical object within the extended reality (XR) environment (Figs. 5-6, 8-13, Pars. 42-43, 50, 52, Wareing’s Figs. 6-7, Pars. 73-75). See claim 3 motivation above.
As to claim 5, Friesen as modified discloses after ceasing display of the remainder of the first plurality of graphical elements, obtaining an indication of a touch input directed to the physical object (Wareing’s Figs. 6-7, Pars. 73-75); and in response to obtaining the indication of the touch input, redisplaying, via the display device the first plurality of graphical elements within the XR environment (Wareing’s Figs. 6-7, Pars. 73-75). See claim 3 motivation above.
As to claim 6, Friesen as modified further discloses prior to displaying the first plurality of graphical elements, obtaining an indication of a touch input directed to the physical object (e.g. expand tool) (Wareing’s Figs. 6-7, Pars. 73-75), and wherein displaying the first plurality of graphical elements within the XR environment includes displaying the first plurality of graphical elements within the XR environment in response to obtaining the indication of the touch input (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69, Wareing’s Figs. 6-7, Pars. 73-75).
As to claim 7, Friesen discloses in accordance with a determination that the first movement of the physical object (108) does not cause the physical object to breach the distance threshold relative to the first graphical element or the second graphical element (stylus does not superimposed/overlapped/breached)(Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69): maintaining an initial output modality as the current output modality for the physical object (108) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69); and maintaining display of the first plurality of graphical elements (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69).
As to claim 8, Friesen discloses in accordance after selecting the first output modality associated with the first graphical element as the current output modality for the physical object (108)(Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69), detecting a subsequent marking input (writing) with the physical object (108) (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69); and in response to detecting the subsequent marking input (writing), displaying, via the display device (104, 240), one or more marks within the XR environment (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69) based on the subsequent marking input and the first output modality (Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69).
As to claim 23, Friesen as modified discloses a respective three-dimensional activation region (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69, Ziraknejad’s Fig. 1B, Par. 38) corresponds to a distance threshold surrounding the first graphical element (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69, Ziraknejad’s Fig. 1B, Par. 38) and the second graphical element, respectively (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69, Ziraknejad’s Fig. 1B, Par. 38). See claim 1 for motivation above.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2011/0205242 to Friesen et al. (Friesen), US 2018/0239515 A1 to Cooper et al. (Cooper), US 2017/0315692 to Wareing et al. (Wareing), and US 2020/0038120 A1 to Ziraknejad et al. (Ziraknejad); in view of US 2022/0276729 A1 to Gutierrez et al. (Gutierrez).
As to claim 9, Friesen as modified does not expressly disclose in response to detecting the subsequent marking input: in accordance with a determination that an input associated with how hard the physical object is being pressed against a physical surface corresponds to a first input value, displaying, via the display device, one or more marks with a first appearance within the XR environment based on the subsequent marking input and the first output modality, wherein the first appearance is associated with a parameter of the one or more marks that corresponds to the first input value; and in accordance with a determination that the input associated with how hard the physical object is being pressed against a physical surface corresponds to a second input value, displaying, via the display device, one or more marks with a second appearance within the XR environment based on the subsequent marking input and the first output modality, wherein the second appearance is associated with the parameter of the one or more marks that corresponds to the second input value.
Gutierrez discloses in response to detecting the subsequent marking input: in accordance with a determination that an input associated with how hard the physical object is being pressed against a physical surface corresponds to a first input value (Figs. 8A-8B, Pars. 93, 100), displaying, via the display device, one or more marks with a first appearance within the XR environment based on the subsequent marking input and the first output modality (Figs. 8A-8B, Pars. 93, 100), wherein the first appearance is associated with a parameter of the one or more marks that corresponds to the first input value (Figs. 8A-8B, Pars. 93, 100); and in accordance with a determination that the input associated with how hard the physical object is being pressed against a physical surface corresponds to a second input value (Figs. 8A-8B, Pars. 93, 100), displaying, via the display device, one or more marks with a second appearance within the XR environment based on the subsequent marking input and the first output modality (Figs. 8A-8B, Pars. 93, 100), wherein the second appearance is associated with the parameter of the one or more marks that corresponds to the second input value (Figs. 8A-8B, Pars. 93, 100).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Friesen with the teaching of Gutierrez to enhance user’s experience as suggested by Gutierrez (Par. 100).
As to claim 10, Friesen as modified discloses the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks within the XR environment (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69; Gutierrez’s Figs. 8A-8B, Pars. 93, 100). See claim 9 motivation above.
Claim(s) 11 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2011/0205242 to Friesen et al. (Friesen), US 2018/0239515 A1 to Cooper et al. (Cooper), US 2017/0315692 to Wareing et al. (Wareing), and US 2020/0038120 A1 to Ziraknejad et al. (Ziraknejad); in view of US 2013/0234967 A1 to Stoddard et al. (Stoddard).
As to claim 11, Friesen as modified does not expressly disclose in response to detecting the subsequent marking input: in accordance with a determination that an input associated with how hard the physical object is being grasped by a user corresponds to a first input value, displaying, via the display device, one or more marks with a first appearance within the XR environment based on the subsequent marking input and the first output modality, wherein the first appearance is associated with a parameter of the one or more marks that corresponds to the first input value; and in accordance with a determination that the input associated with how hard the physical object is being grasped by the user corresponds to a second input value, displaying, via the display device, one or more marks with a second appearance within the XR environment based on the subsequent marking input and the first output modality, wherein the second appearance is associated with the parameter of the one or more marks that corresponds to the second input value.
Stoddard discloses in response to detecting the subsequent marking input: in accordance with a determination that an input associated with how hard the physical object (stylus) is being grasped by a user corresponds to a first input value (e.g. light grip), displaying, via the display device, one or more marks with a first appearance within the display environment based on the subsequent marking input and the first output modality (Figs. 1-3, Par. 7), wherein the first appearance is associated with a parameter of the one or more marks that corresponds to the first input value (light grip corresponds to narrow line drawn)(Figs. 1-3, Par. 7); and in accordance with a determination that the input associated with how hard the physical object is being grasped by the user corresponds to a second input value (e.g. tighter grip), displaying, via the display device, one or more marks with a second appearance within the display environment based on the subsequent marking input and the first output modality (Figs. 1-3, Par. 7), wherein the second appearance is associated with the parameter of the one or more marks that corresponds to the second input value (tighter grip corresponds to thicker line drawn) (Figs. 1-3, Par. 7).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Friesen with the teaching of Stoddard to mimic a real writing device thereby provide an intuitive inputting process as suggested by Stoddard (Par. 7).
As to claim 12, Friesen as modified wherein the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks within the XR environment (Friesen’s Figs. 1-6, Pars. 71-72, 74, see also Pars. 66-69, Stoddard’s Figs. 1-3, Par. 7). See claim 11 for motivation above.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-2, 4-12, 17, 19-20, and 22-23 have been considered but are moot in view of new ground(s) of rejection.
Examiner notes that the new claim elements are now addressed by reference Cooper as necessitated by amendments. Please see above for full basis of rejection.
On pages 12-15 of the Applicant’s Remarks, the Applicant argues that Cooper does not teach the amended limitation of the extended reality (XR) environment includes a representation of the physical object with a tip" or "maintaining display of the first [or second] graphical element overlaid on the tip of the representation of the physical object within the extended reality (XR) environment". The Examiner respectfully disagrees. See the new rejection above, as necessitated by amendments, for detail.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2018/0239515 A1 to Cooper et al. teaches an apparatus with a virtual controller and a graphical representation of a physical controller in computer-generated environment.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JARURAT SUTEERAWONGSA whose telephone number is (571)270-7361. The examiner can normally be reached Monday thru Thursday, 8:30AM to 6:00PM, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lun Yi Lao can be reached at 571-272-7671. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JARURAT SUTEERAWONGSA/Examiner, Art Unit 2621
/LUNYI LAO/Supervisory Patent Examiner, Art Unit 2621