DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is in reply to an application and a preliminary amendment both filed on January 7, 2025 regarding Application No. 18/992,030. Claims 1-13 are pending.
Priority
The present application is the National Phase of PCT International Application No. PCT/KR2022/010406 filed on July 15, 2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on January 7, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the Office. Please note that the Office has included the application number and art unit number on the IDS. It is noted that PCT/ISA/237 (4 pages) and PCT/ISA/210 (6 pages) forms for PCT/KR2022/010406 were also submitted on January 7, 2025 but not listed on the IDS.
Specification
The use of the term(s) WI-FI”, “BLUETOOTH”, “LTE”, and “3GPP”, which is a trade name(s) or a mark(s) used in commerce, has been noted in this application. The term(s) should be accompanied by the generic terminology; furthermore the term(s) should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Objections
Claims 1-13 are objected to for the reasons discussed below.
Claims 1, 7, and 13: “XR device” (line 1 of each claim) may need to be changed to “extended reality (XR) device” to identify the term corresponding to “XR”.
Also, “VR image” (claim 1: line 8; claim 7: 4th to the last line; claim 13: 3rd to the last line) may need to be changed to “virtual reality (VR) image” to identify the term corresponding to “VR”.
Claim 2: “an interaction-related operation for a target object of the VR image” should be changed to “[[an]]the interaction-related operation for [[a]]the target object of the VR image” since the terms were previously recited.
Claim 3: “ToF camera” may need to be changed to “time of flight (ToF) camera” to identify the term corresponding to “ToF”.
Claim 5: “a virtual object” (3rd to the last line) should be changed to “[[a]]the virtual object” since the term was previously recited.
Also, “the VR display” (2nd to the last line) should be changed to “the [[VR ]]display” to correspond to the term previously recited.
Claim 7: “a change in extension or contraction of the connected wire” (lines 8-9) should be changed to “[[a]]the change in extension or contraction of the connected wire” since the claim language was previously recited.
Claim 9: “a motion of bending the finger” (3rd to the last line) should be changed to “[[a]]the motion of bending the finger” since the claim language was previously recited.
Also, “a motion of extending the finger” (last line) should be changed to “[[a]]the motion of extending the finger” since the claim language was previously recited.
Claims 11-12: “an operation of gripping a target object by a virtual object of the VR image” (lines 2-3 of each claim) should be changed to “[[an]]the operation of gripping [[a]]the target object by [[a]]the virtual object of the VR image” since the claim language was previously recited.
Claims 2-6 and 8-12: these claims depend from an objected to claim.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“communication module” in claims 1 and 7: interpreted as corresponding to Wi-Fi, Bluetooth, or LTE using 3GPP communication methods/standards. ([0048] of the amended Specification).
“output module” in claims 7 and 10-12: interpreted as a motor. ([0101] and [0153]).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If Applicants do not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, Applicants may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicants are advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Motta et al. in US 2018/0082482 A1 (hereinafter Motta).
Regarding claim 1, Motta teaches:
A mixed reality device (2000 in FIG. 4) comprising (Motta: FIG. 4 and “[0044]... [A] mixed reality system 1900 may include a HMD 2000 such as a headset, helmet, goggles, or glasses....”, see also FIGs. 1-3, [0018], [0039], and [0050]):
a communication module (2040 in FIG. 4) that transmits and receives data to and from a controller (2100) (Motta: FIG. 4 and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, see also “[0039]... [D]etected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space.... [T]he user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”,);
a first camera (2050 in FIG. 4, see also 210-213 in FIGs. 2A-C) for collecting external environment information and tracking a user's position and a second camera (2060 in FIG. 4, see also 217 in FIGs. 2A-C) for tracking a position of a hand (294 in FIG. 2A) of the user (USER in FIG. 4, see also 290 in FIG. 2A) wearing the controller (Motta: FIGs. 2A and 4, “[0049]... [T]he HMD 2000 [in FIG. 4] may include world sensors 2050 that collect information about the user's environment (video, depth information, lighting information, etc.), and user sensors 2060 that collect information about the user (e.g., the user's... hand gestures...)....World and user sensors of an example HMD are shown in FIGS. 2A through 2C.”, see also FIGs. 1 and 2B-C, “[0023]... [Referring to FIGs. 2A-2C,] a HMD 200 may include world sensors 210-213 that collect information about the user 290's environment (video, depth information, lighting information, etc.), and user sensors 214-217 that collect information about the user 290 (e.g., the user's... hand gestures, etc.)...”, [0024] (including: “[T]he world sensors [210-213] may be used to provide orientation and motion information for the user in the real environment....”, “[0029]... [T]he world sensors may include one or more “head pose” sensors 212 (e.g., IR or RGB cameras) that may capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment. The information collected by sensors 212 may, for example, be used to augment information collected by an inertial-measurement unit (IMU) of the HMD 200....”, and “[0039]... [T]he user sensors may include one or more hand sensors 217 (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms....”);
a display (screens in FIG. 4, see also 202 in FIG. 2A) that displays a virtual object corresponding to the hand on a VR image corresponding to the collected external environment information (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C]... that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0044]... [T]he HMD 2000 may include a near-eye VR projector that projects frames including left and right images on screens that are viewed by a user.... As another example, the HMD 2000 may include a direct retinal projector that scans frames including left and right images, pixel by pixel, directly to the user's eyes....”, and “[0050] The HMD 2000 may be configured to render and display frames to provide a 3D virtual view 2002 for the user at least in part according to world sensor 2050 and user sensor 2060 inputs. The virtual space 2002 may include renderings of the user's environment, including renderings of real objects 2012 in the user's environment, based on video captured by one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality, high-resolution video of the user's environment in real time for display. The virtual space 2002 may also include virtual content (e.g., virtual objects, 2014, virtual tags 2015 for real objects 2012, avatars of the user, etc.) generated by the mixed reality system 1900 and composited with the projected 3D view of the user's real environment. FIG. 3 describes an example method for collecting and processing sensor inputs to generate content in a 3D virtual view 2002 that may be used in a mixed reality system 1900 as illustrated in FIG. 4, according to some embodiments.”, see also FIGs. 1-2A, “[0019]... HMD 100 [in FIG. 1] may be a near-eye VR system that projects left and right images on screens in front of the user 190's eyes that are viewed by a subject.... As another example, HMD 100 may be a direct retinal projector system that scans left and right images, pixel by pixel, to the subject's eyes....”, and “[0023] As shown in FIGS. 2A through 2C, HMD 200 may be worn on a user 290's head so that the projection system displays 202 (e.g. screens and optics of a near-eye VR system, or reflective components (e.g., ellipsoid mirrors) of a direct retinal projector system) are disposed in front of the user 290's eyes 292....”); and
a processor (2030 in FIG. 4) that tracks the position of the hand and processes the virtual object to move on the VR image so as to display the processed virtual object on the display (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C]... that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0045]... HMD 2000 may also include a controller 2030 configured to implement functionality of the mixed reality system 1900 as described herein and to generate the frames (each frame including a left and right image) that are projected or scanned by the 3D projector 2020 into the 3D virtual view 2002.... HMD 2000 may also include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection..... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, [0046] (including “... Controller 2030 may include central processing units (CPUs) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in that instruction set architecture....”, and “[0050] The HMD 2000 may be configured to render and display frames to provide a 3D virtual view 2002 for the user at least in part according to world sensor 2050 and user sensor 2060 inputs. The virtual space 2002 may include renderings of the user's environment, including renderings of real objects 2012 in the user's environment.... The virtual space 2002 may also include virtual content (e.g., virtual objects, 2014, virtual tags 2015 for real objects 2012, avatars of the user, etc.) generated by the mixed reality system 1900 and composited with the projected 3D view of the user's real environment. FIG. 3 describes an example method for collecting and processing sensor inputs to generate content in a 3D virtual view 2002 that may be used in a mixed reality system 1900 as illustrated in FIG. 4, according to some embodiments.”),
wherein the processor receives operation detection data corresponding to a motion of the hand as input from the controller through the communication module, and processes the virtual object to perform an interaction-related operation with a target object of the VR image based on the received input so as to display the processed virtual object on the display (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C]... that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.” , “[0045]... HMD 2000 may also include a controller 2030 configured to implement functionality of the mixed reality system 1900 as described herein and to generate the frames (each frame including a left and right image) that are projected or scanned by the 3D projector 2020 into the 3D virtual view 2002.... HMD 2000 may also include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection..... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, and “[0050] The HMD 2000 may be configured to render and display frames to provide a 3D virtual view 2002 for the user at least in part according to world sensor 2050 and user sensor 2060 inputs. The virtual space 2002 may include renderings of the user's environment, including renderings of real objects 2012 in the user's environment.... The virtual space 2002 may also include virtual content (e.g., virtual objects, 2014, virtual tags 2015 for real objects 2012, avatars of the user, etc.) generated by the mixed reality system 1900 and composited with the projected 3D view of the user's real environment. FIG. 3 describes an example method for collecting and processing sensor inputs to generate content in a 3D virtual view 2002 that may be used in a mixed reality system 1900 as illustrated in FIG. 4, according to some embodiments.”).
However, it is noted that Motta does not teach:
An XR device,
but which would have been obvious to include, such that Motta as modified teaches: An XR device comprising (mixed reality device of Motta combined with an XR device as discussed):, since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to display virtual images.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Gonzalez Franco et al. in US 2020/0278740 A1 (hereinafter Gonzalez Franco).
Regarding claim 2, Motta as modified teaches:
The XR device of claim 1.
However, it is noted that Motta as modified does not teach:
wherein the processor controls the display to display an interaction-related operation for a target object of the VR image while a feedback signal corresponding to the operation detection data is generated from the controller.
Gonzalez Franco teaches:
wherein a processor (of 102 in FIG. 1) controls a display (of 106) to display an interaction-related operation for a target object (602 in FIG. 6A) of a VR image while a feedback signal (vibration signal) corresponding to operation detection data is generated from a controller (108 in FIG. 1) (Gonzalez Franco: FIGs. 1 and 6A, “[0020]... The virtual reality system 100 [in FIG. 1] may include a base station 102. The base station 102 can include hardware and/or software for generating and executing a virtual reality world, including receiving and processing inputs from a user 104, and generating and outputting feedback to the user 104. The base station 102 may be any computing device, including a personal computer (PC), server, gaming console,... simulator, etc.”, “[0021]... The headset 106 [in the virtual reality system 100] may be... a head-mounted display (HMD) that receives visual information from the virtual reality world being executed by the base station 102 and includes a display for displaying the visual information to the user 104....”, “[0022]... [T]he controller 108 [in the virtual reality system 100] may be a handheld device that the user 104 can hold in her hand 110 and manipulate with her fingers to provide inputs to the base station 102. The controller 108 may include sensors 112 capable of detecting finger motions and/or forces.... Furthermore, the controller 108 may receive haptic information from the virtual reality world being executed by the base station 102 and can render the haptic information to the user's hand 110 via vibrators 114....”, “[0062] The base station 102 may generate visual information that simulates the virtual fingers 604 moving in the direction of the arrows 606 shown in FIG. 6A and grasping the virtual object 602. This visual information may be transmitted by the base station 102 to the display 514 and rendered to the user 104....”, and “[0063]... [T]he base station 102 may also generate haptic information that simulates the virtual fingers 604 grasping the virtual object 602.... [T]he characteristics of the haptic information, including amplitude, frequency, and/or duration, may depend on several factors, such as the level of force applied by the fingers 206... and the virtual size, shape, weight, texture, structure, material, and/or orientation of the virtual object 602. For example, in a scenario where the virtual object 602 is a potato chip, if the user's fingers 206 apply gentle forces... when handling the virtual potato chip, then the base station 102 may generate little to no haptic feedback. But if the user's fingers 206 squeeze... too hard, then the base station 102 may simulate the virtual potato chip cracking or breaking, and generate high-frequency vibrations as haptic feedback, thus rendering a crisp sensation to the user's fingers 206. The haptic information may be transmitted by the base station 102 to the controller 108 and rendered to the user's fingers 206 via the actuators 306....”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Gonzalez Franco, such that Motta as modified teaches: wherein the processor controls the display to display an interaction-related operation for a target object of the VR image while a feedback signal corresponding to the operation detection data is generated from the controller (processor, display, interaction-related operation, target object, VR image, operation detection data, and controller of Motta as modified combined with the processor, display, interaction-related operation, target object, VR image, feedback signal, operation detection data, and controller of Gonzalez Franco), to provide a realistic user experience. (Gonzalez Franco: “[0063]... [T]he combination of contemporaneous visual feedback and haptic feedback may create compelling realistic experience for the user 104....”).
Claim 3-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Yamano et al. in US 2023/0095328 A1 (hereinafter Yamano).
Regarding claim 3, Motta as modified teaches:
The XR device of claim 1, wherein the first camera is a general camera (Motta: “[0025]... [T]he world sensors may include one or more “video see through” cameras 210 (e.g., RGB (visible light) video cameras) that capture high-quality video of the user's environment that may be used to provide the user with a virtual view of their real environment....”, see also “[0028] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be one world mapping sensor 211 located on a front surface of the HMD 200.... [A] world mapping sensor 211 may include an IR light source and IR camera....”, “[0029]... [T]he world sensors may include one or more “head pose” sensors 212 (e.g., IR or RGB cameras) that may capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment....”, “[0031]... [T]he world sensors may include one or more light sensors 213 (e.g., RGB cameras) that capture lighting information (e.g., direction, color, and intensity) in the user's environment that may, for example, be used in rendering virtual content in the virtual view of the user's environment, for example in determining coloring, lighting, shadow effects, etc. for virtual objects in the virtual view....”).
However, it is noted that Motta as modified does not teach:
the second camera is a ToF camera.
Yamano teaches:
a second camera (121 in FIG. 5) is a ToF camera (Yamano: FIG. 5 and “[0095] The outward camera 121... is installed to image the outside of the AR glasses.... The outward camera 121 can image an operation using the user's fingers.... [T]he outward camera 121 may... include... a ToF camera....”, see also FIGs. 4 and 6-7).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Yamano, such that Motta as modified teaches: wherein the first camera is a general camera and the second camera is a ToF camera (first and second cameras and general camera of Motta combined with the second camera and ToF camera of Yamano), to track a position of a user’s hand.
Regarding claim 4, Motta as modified by Yamano teaches:
The XR device of claim 3, wherein the XR device is a head-mounted display (HMD) device type (HMD 2000 in FIG. 4 of Motta) that can be worn on the user's head, and wherein, when worn, the first camera is positioned above both eyes of the user to scan a forward environment (Motta: i.e., positioned as claimed; FIGs. 2A-C and 4 and “[0026] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be two video see through cameras 210A and 210B located on a front surface of the HMD 200 at positions that are substantially in front of each of the user 290's eyes 292A and 292B. However, in various embodiments, more or fewer cameras 210 may be used in a HMD 200 to capture video of the user 290's environment, and cameras 210 may be positioned at other locations....”, see also FIG. 1, “[0028] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be one world mapping sensor 211 located on a front surface of the HMD 200. However,... world mapping sensor 211 may be positioned at other locations....”, “[0030] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be two head pose sensors 212A and 212B located on a front or top surface of the HMD 200....”, and “[0032] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be one light sensor 213 located on a front or top surface of the HMD 200....”), and
the second camera is positioned below the both eyes of the user to track the hand of the user wearing the controller (Motta: FIGs. 2A-C and “[0040] As shown in the non-limiting example HMD 200 of FIGS. 2A through 2C,... there may be one hand sensor 217 located on a bottom surface of the HMD 200....”).
Regarding claim 6, Motta as modified teaches:
The XR device of claim 1.
However, it is noted that Motta as modified does not teach:
wherein the received input is either one of a first input and a second input,
wherein the first input is operation detection data according to a motion of bending a hand wearing the controller, and the second input is operation detection data according to a motion of extending the hand, and
wherein the processor displays an operation that maintains the interaction when the first input is received and displays a motion that leaves the interaction when the second input is received.
Yamano teaches:
wherein received input is either one of a first input (corresponding to a bending motion - gripping) and a second input (corresponding to an extending motion - ungripping) (Yamano: FIGs. 23-26, “[0145]... [A]s illustrated in FIG. 23, when the user performs a virtual object picking operation by bringing the fingertips of the thumb and the forefinger into contact with each other while the virtual gripping point exists in the grip detection region, gripping processing is performed. The display of the virtual object gripped by the user may be changed. In an example illustrated in FIG. 23, the virtual object in the gripped state is highlighted.”, and “[0147] When the user releases the virtual object gripping operation by separating the fingertips of the thumb and the forefinger from each other, the virtual object can freely move without restricting the relative positional and postural relationship of the virtual object with the user's hand. For example, when the user releases the virtual object gripping operation in the state illustrated in FIG. 25, the virtual object falls due to gravity as illustrated in FIG. 26....”, see also FIGs. 21-22, “[0140] FIGS. 21 to 26 sequentially illustrate a flow of gripping a virtual object.... Here, it is assumed that a user grips a virtual object placed on a desk using a hand on which the controller 110 is worn. Note that the desk may be either a real object or a virtual object. The flow of gripping the virtual object illustrated in FIGS. 21 to 26 is performed by the application execution unit 1201 on the basis of the information on a position of a user's hand and a user's finger gesture acquired through the controller 110, and switching in displaying a virtual object is also performed according to a user's virtual object gripping operation.” and “[0146] In a state where the user is gripping the virtual object while maintaining the contact between the fingertips of the thumb and the forefinger, the relative positional and postural relationship between the user's hand and the virtual object is maintained. As illustrated in FIGS. 24 and 25, when the user changes the posture of the hand gripping the virtual object or lifts the hand gripping the virtual object, the posture of the virtual object is also changed or the virtual object is also lifted to follow the user's hand.”),
wherein the first input is operation detection data according to a motion of bending a hand wearing a controller (110 in FIG. 7), and the second input is operation detection data according to a motion of extending the hand (Yamano: FIGs. 23-26, “[0140] FIGS. 21 to 26 sequentially illustrate a flow of gripping a virtual object.... Here, it is assumed that a user grips a virtual object placed on a desk using a hand on which the controller 110 is worn....”, and [0145]-[0147], see also FIGs. 1-5 and 19), and
wherein a processor (140 in FIGs. 7 and 12) displays an operation that maintains the interaction when the first input is received and displays a motion that leaves the interaction when the second input is received (Yamano: FIGs. 7, 12, and 25-26, “[0084]... [C]ontrol unit 140 generally controlling an overall operation of the AR system 100.... The control unit 140 executes various programs read out from the storage unit 150 to perform various processes.”, “[0146] In a state where the user is gripping the virtual object while maintaining the contact between the fingertips of the thumb and the forefinger, the relative positional and postural relationship between the user's hand and the virtual object is maintained....”, and [0147], see also FIGs. 5 and 24 and [0117]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Yamano, such that Motta as modified teaches: wherein the received input is either one of a first input and a second input (input of Motta as modified combined with the input and first and second inputs of Yamano), wherein the first input is operation detection data according to a motion of bending a hand wearing the controller, and the second input is operation detection data according to a motion of extending the hand (operation detection data, hand, and controller of Motta as modified combined with the first and seconds inputs, operation detection data according to a motion of bending a hand wearing the controller, and operation detection data according to a motion of extending the hand), and wherein the processor displays an operation that maintains the interaction when the first input is received and displays a motion that leaves the interaction when the second input is received (processor and input of Motta as modified combined with the processor, operation that maintains the interaction, first and second inputs, and motion that leaves the interaction of Yamano), so a user can interact with a virtual target object by gripping and ungripping the virtual target object.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Yamano, in further view of Balan et al. in US 2017/0357332 A1 (hereinafter Balan).
Regarding claim 5, Motta as modified by Yamano teaches:
The XR device of claim 4.
However, it is noted that Motta as modified by Yamano does not teach:
wherein the processor controls a point of the hand of the user wearing the controller to be captured through the second camera, controls continuous images including a depth map of the user's hand to be acquired based on the captured point, and operates a virtual object corresponding to the hand to be displayed on the VR display based on the acquired continuous images.
Ballan teaches:
wherein a processor controls a point of a hand (38 in FIG. 8) of a user (26) wearing a controller (40 in FIG. 6) to be captured through a second camera (18 in FIG. 1), controls continuous images including a depth map of the user's hand to be acquired based on the captured point, and operates a virtual object corresponding to the hand to be displayed on a VR display based on the acquired continuous images (Balan: wearing controller 40 using a finger loop; FIGs. 1-2, 6, and 8-9, “[0020] FIG. 1 schematically illustrates an example of a HMD device 10. FIGS. 2-5 are illustrations of the Microsoft Hololens, which represents one recent embodiment of a HMD.”, “[0030] One example of a HMD is the Microsoft Hololens, which is a pair of mixed reality head-mounted smartglasses....”, “[0036] In addition to a central processing unit (CPU) and graphics processing unit (GPU), Hololens features a custom-made Microsoft Holographic Processing Unit (HPU), a coprocessor manufactured specifically for the Hololens. The main purpose of the HPU is processing and integrating data from the sensors, as well as handling tasks such as spatial mapping, [and] gesture recognition.... The HPU processes terabytes of information from the Hololens's sensors from real-time data.”, “[0041]... [T]he Hololens includes a depth camera, which is capable of detecting the 3D location of objects located within the depth camera's FOV. Technical details of exactly how the depth camera accomplishes such detection are known to those skilled in the art, but are not necessary for the present disclosure. Suffice it to say that the depth camera is able to accurately detect, on a pixel-by-pixel basis, the exact 3D location of each point on a physical object within the camera's field of view. While the Hololens uses a depth camera, stereoscopic optics can also be used to detect the distance of objects from the HMD and the locations of such objects in 3D space via triangulation. In either event, such sensors can detect the 3D location (x, y and z coordinates) of real objects located within the FOV relative to the HMD. In the case of a Controller, the depth camera of the HMD can be used to detect the 3D location of the Controller relative to the HMD.”, “[0043] (including: ... [T]he Hololens has the ability to track the movement of a user's hands through space and to identify and interpret a variety of hand poses, gestures and movements to manipulate virtual objects in the AR space....”, “[0044]... [T]he headset [of Hololens] uses Bluetooth LE to pair with a Controller, called a “Clicker,” a thumb-sized finger-operating input device that can be used to enable the user to select, scroll, hold, and double-click to interact with virtual objects within the augmented reality space.”, “[0045]... Controller 40 can also include one more momentary switch(es) 48 for selective activation by the user to control a virtual cursor and/or to manipulate virtual objects in various ways (such as, for example, select, move, rotate, scroll, etc.). Controller 40 can also include an elastic finger loop (for holding the device)....”, “[0051]... [I]n the example illustrated in FIG. 8, the HMD device 10 worn by the user 26 may be configured to detect motion of the user's hand. Based on a series of images captured by the optical sensor system 16, the HMD device 10 may determine whether motion of hand 38 of the user 26 is trackable. For example, the user's hand at positions 38 and 38A are within the field of view of the optical sensor system 16. Accordingly, motion of the user's hand moving from position 38 to position 38A over time T1 is trackable by the HMD device 10.... It will be appreciated that the user's hand is determined to be trackable by the HMD when the HMD can monitor the hand for gesture input. Thus, the user's hand is deemed to be trackable, for example, when computer algorithms implemented in software executed on the processor of the HMD device 10 identify the hand in images captured by the onboard camera and begin tracking the hand, until a point in time at which those algorithms lose track of the hand. Techniques that may be used to track the hand the hand include searching for regions of similar color values and segmenting a portion of the image based on the color values from the rest of the image, as well as searching for regions of pixels that have changed, indicating foreground movement by a hand or other object. When depth information is available, the hand may be located using skeletal tracking techniques in addition or as an alternative to the above....”, “[0052]... [T]he HMD device 10 communicates to the user whether motion of the user's hand is trackable.... [I]n response to at least determining that motion of the hand is trackable, the HMD device 10 modifies the visual appearance of the holographic cursor to indicate that motion of the hand is trackable. In the example illustrated in FIG. 8, the visual appearance of the holographic cursor is modified to appear as holographic cursor 28, which is an unfilled circle. Accordingly, as the user moves the hand from position 38 to position 38A over time T1, the user is shown holographic cursor having visual appearance 28 and is thus provided with the feedback that motion of the user's hand is currently trackable, and any hand gestures or hand movements will be tracked by the HMD device 10.”, and “[0064] For Hololens, the hand tracking sensor consists of a depth camera that observes the hands moving through space. The depth image can be used to segment the hand from the background and the rest of the body, classify pixels as belonging to different hand parts using decision trees/jungles, and compute centroids for them (palm, fingertips, etc.) in 3D space.”, see also “[0023]... [T]he outward facing optical sensor 18 may include one or more component sensors, including an RGB camera and a depth camera. The RGB camera may be a high definition camera or have another resolution. The depth camera may be configured to project non-visible light, such as infrared (IR) radiation, and capture reflections of the projected light, and based thereon, generate an image comprised of measured depth data for each pixel in the image. This depth data may be combined with color information from the image captured by the RGB camera, into a single image representation including both color data and depth data, if desired. In a virtual reality configuration, the color and depth data captured by the optical sensor system 16 may be used to perform surface reconstruction and generate a virtual model of the real world background that may be displayed to the user via the display 12. Alternatively, the image data captured by the optical sensor system 16 may be directly presented as image data to the user on the display 12.”, “[0027] Additionally, the optical sensor information received from the optical sensor system 16 may be used to identify and track objects in the field of view of optical sensor system 16. For example, depth data captured by optical sensor system 16 may be used to identify and track motion of a user's hand. The tracked motion may include movement of the user's hand in three-dimensional space, and may be characterized with six degrees of freedom.... The tracked motion may also be used to identify and track a hand gesture made by the user's hand. For example, one identifiable hand gesture may be moving a forefinger upwards or downwards. It will be appreciated that other methods may be used to identify and track motion of the user's hand. For example, optical tags may be placed at known locations on the user's hand or a glove worn by the user, and the optical tags may be tracked through the image data captured by optical sensor system 16.”, “[0048] At this point it bears repeating that the foregoing description of the Hololens and Clicker are provided merely as examples of a system and environment in which the systems and methods disclosed and/or claimed herein can be implemented. The inventions described herein can be readily adapted to other HMD devices and AR and VR systems/environments, as well as other traditional computing systems and environments, including other conventional display devices.”, and [0058]).
Before the effective filing date of the claimed invention, it would have been obvious to include: the features taught by Balan, such that Motta as modified teaches: wherein the processor controls a point of the hand of the user wearing the controller to be captured through the second camera, controls continuous images including a depth map of the user's hand to be acquired based on the captured point, and operates a virtual object corresponding to the hand to be displayed on the VR display based on the acquired continuous images (processor, hand of the user, controller, second camera, virtual object, and display of Motta as modified combined with the processor, point of a hand of a user, controller, second camera, continuous images, depth map, virtual object, and display of Balan; also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use for control motion input), for control motion input.
Claims 7 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Lee et al. in KR 10-2017-0008498 A (hereinafter Lee; January 7, 2025 IDS reference – an original copy and a translation of the abstract was submitted with the IDS; an original copy and a full machine translation thereof is/was provided with the first Office action mailed in response to the filing of the instant application and preliminary amendment).
Regarding claim 7, Motta teaches:
A controller apparatus (2100 in FIG. 4) for a mixed reality device (2000 in FIG. 4), the controller apparatus comprising (Motta: FIG. 4, “[0044]... [A] mixed reality system 1900 may include a HMD 2000 such as a headset, helmet, goggles, or glasses....”, and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, see also FIGs. 1-3, [0018], [0039], and [0050]):
a communication module (of 2100) that transmits and receives data when connected to the mixed reality device (Motta: FIG. 4 and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”);
transmits operation detection data corresponding to input from the controller apparatus to the mixed reality device, and a hand of a user being recognized through a camera (2060 in FIG. 4, see also 217 in FIGs. 2A-C) of the mixed reality device so as to interact with a VR image displayed on the mixed reality device (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0049]... [T]he HMD 2000 [in FIG. 4] may include... user sensors 2060 that collect information about the user (e.g., the user's... hand gestures...). The sensors... 2060 may provide the collected information to the controller 2030 of the mixed reality system 1900.... [U]ser sensors of an example HMD are shown in FIGS. 2A through 2C.”, and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, see also FIGs. 1-2C); and
an output module (2020 in FIG. 4) that outputs a feedback signal (visual feedback signal) when an interaction with the VR image is performed based on the operation detection data (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, and “[0045] HMD 2000 may include a 3D projector 2020 that implements the VR projection technology that generates the 3D virtual view 2002 viewed by the user... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, see also FIGs. 1-2A).
However, it is noted that Motta does not teach:
an XR device,
but which would have been obvious to include since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to display virtual images.
However, it is noted that Motta as modified does not teach:
a body configured to be worn on a user's hand;
a sensor that detects a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the body;
a processor that transmits operation detection data corresponding to a change in extension or contraction of the connected wire to the XR device based on the hand of the user wearing the body being recognized through a camera of the XR device so as to interact with a VR image displayed on the XR device.
Lee teaches:
a controller apparatus (200 in FIG. 2) comprising (Lee: FIG. 2 and “[0088] Electronic device 200 related to the embodiment of the present invention, the display unit 151, the wearable unit 201, tension (tension) changes in accordance with the movement of the finger which is provided in the center that covers the back of the hand of the wearable unit 201 sensing unit processes the function is set to determine the movement of the finger, and the movements of the determined finger based on the change in the tension of the wearable unit 201 is detected includes 140, via the sensing unit 140 to sense the and it may include a controller 180 that displays the screen on the display unit 151.”, see also FIGs. 3-16, “[0092] The controller 180 may be provided in a form embedded in the inside of the back plate of the wearable unit of the display unit 151 (201). Controller 180 to the sensing unit display portion 151 determines the movement of the finger, and processes the function is set to the motion of the determined finger based on the change in the tension of the wearable unit 201 is detected through a 140 It may display a screen.”, and “[0098] Controller 180 if more than the sensed tension and the reference value is set by comparing the detection result of the tension sensor group (143-1, 143-2), the reference value is the tension sensor detects a finger being moved may be determined. Thus, on the basis of the data stored in the memory 170, in accordance with the detection result of the first tension sensor (143-1) and the second tension sensor (143-2), to unlock the screen of the display section 151 or, It can display and select a menu, and the user may execute the selected application.”):
a body (201 in FIG. 2) configured to be worn on a user's hand (Lee: FIG. 2 and “[0086] Electronic device 200... may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”);
a sensor (140 in FIG. 3) that detects a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the body (Lee: FIGs. 2-3 and “[0090] Sensing unit 140 detects a tension change in the wearable unit 201 provides the detection result to the controller 180. Here, the tension (tension) may refer to the force applied to the line. Wearing the wearable unit 201 and the finger piece state can be measured by the tensile force is zero, when the fingers are folded to the tension generated in the wearable unit 201. Thus, the sensing unit 140 may provide the sensed value to the controller 180 when the tension of the wearable unit 201 changes. The sensing unit 140 may be provided in a region easy to detect tension changes in accordance with the finger motion.”);
a processor (180 in FIG. 3) that transmits operation detection data corresponding to a change in extension or contraction of the connected wire to a display device (151) so as to interact with an image displayed on the display device (Lee: FIG. 3 and “[0092] The controller 180 may be provided in a form embedded in the inside of the back plate of the wearable unit of the display unit 151 (201). Controller 180 to the sensing unit display portion 151 determines the movement of the finger, and processes the function is set to the motion of the determined finger based on the change in the tension of the wearable unit 201 is detected through a 140 It may display a screen.”, see also FIGs. 2 and 4-16, “[0020] Herein from the description that the electronic apparatus include wearable devices (wearable device, for example,... glass-type terminals (smart glass), HMD (head mounted display)), etc. should be included can is.”, [0031], [0043], [0083], [0086], and “[0098] Controller 180 if more than the sensed tension and the reference value is set by comparing the detection result of the tension sensor group (143-1, 143-2), the reference value is the tension sensor detects a finger being moved may be determined. Thus, on the basis of the data stored in the memory 170, in accordance with the detection result of the first tension sensor (143-1) and the second tension sensor (143-2), to unlock the screen of the display section 151 or, It can display and select a menu, and the user may execute the selected application.”); and
an output module (150 in FIG. 1) that outputs a feedback signal (visual feedback signal) when an interaction with the image is performed based on the operation detection data (Lee: FIGs. 1-2 and 7, “[0023] The electronic device 100 includes... an output unit 150....”, “[0028] Output unit 150 is for generating an output related to visual, auditory or tactile sense, a display unit 151,the audio output unit 152, haeptip module 153, a light output portion 154 of the at least one can do....”, “[0086] Electronic device 200... may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”, “[0111] If the user does not take any action in a state wearing the wearable unit 201 in the hand, that is, when the input has not occurred the controller 180 may be a display unit 151 may display a lock (LOCK) screen . Thus, when the user fold the index finger and ring finger at the same time the first tension value (TS_1) and the second tension value (TS_2) is input at the same time as the controller 180.”, and “[0112] The controller 180 may display a first tension value (TS_1) and the second tension value if found to be (TS_2) is greater than the reference value is pre-set, the secondary display a user input standby state in the standby screen 151.”, see also FIG. 1 (haptic module 153), FIGs. 8-16 and [0067]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Lee, such that Motta as modified teaches: A controller apparatus for an XR device, the controller apparatus comprising (controller apparatus and XR device of Motta as modified combined with the controller apparatus of Lee): a body configured to be worn on a user's hand (as taught by Lee); a communication module that transmits and receives data when connected to the XR device (as taught by Motta as modified); a sensor that detects a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the body (as taught by Lee); a processor that transmits operation detection data corresponding to a change in extension or contraction of the connected wire to the XR device based on the hand of the user wearing the body being recognized through a camera of the XR device so as to interact with a VR image displayed on the XR device (operation detection data, input, XR device, hand of a user, camera, and VR image of Motta as modified combined with the processor that transmits of Lee; it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use for control input); and an output module that outputs a feedback signal when an interaction with the VR image is performed based on the operation detection data (output module of Motta as modified combined with the output module of Lee), for user interaction with a VR image using a controller apparatus.
Regarding claim 13, Motta teaches:
An operating method of a mixed reality device (2000 in FIG. 4), the method comprising (Motta: FIG. 4, “[0044]... [A] mixed reality system 1900 may include a HMD 2000 such as a headset, helmet, goggles, or glasses....”, and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, see also FIGs. 1-3, [0018], [0039], and [0050]):
connecting the mixed reality device and a controller apparatus (2100 in FIG. 4) (Motta: FIG. 4 and “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”);
recognizing a hand of a user through a camera (2060 in FIG. 4, see also 217 in FIGs. 2A-C) of the mixed reality device (Motta: FIG. 4, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, and “[0049]... [T]he HMD 2000 [in FIG. 4] may include... user sensors 2060 that collect information about the user (e.g., the user's... hand gestures...). The sensors... 2060 may provide the collected information to the controller 2030 of the mixed reality system 1900.... [U]ser sensors of an example HMD are shown in FIGS. 2A through 2C.”, see also FIGs. 1-2C);
detecting an input from the controller apparatus (Motta: “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.” and “[0045]... HMD 2000 [in FIG. 4] may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”); and
displaying an interaction-related operation on a VR image displayed on the mixed reality device in response to receiving operation detection data corresponding to the input from the controller apparatus (Motta: “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.” and “[0045]... HMD 2000 [in FIG. 4] may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”).
However, it is noted that Motta does not teach:
an XR device,
but which would have been obvious to include since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to display virtual images.
However, it is noted that Motta as modified does not teach:
the controller apparatus worn on a user’s hand;
the hand of the user wearing the controller apparatus;
detecting a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the controller apparatus; and
the operation detection data corresponding to the detected change in the extension or contraction of the wire.
Lee teaches:
connecting a display device (151 in FIG. 2) and a controller apparatus (200) worn on a user's hand (Lee: FIG. 2, “[0086] Electronic device 200... may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”, “[0088] Electronic device 200 related to the embodiment of the present invention, the display unit 151, the wearable unit 201, tension (tension) changes in accordance with the movement of the finger which is provided in the center that covers the back of the hand of the wearable unit 201 sensing unit processes the function is set to determine the movement of the finger, and the movements of the determined finger based on the change in the tension of the wearable unit 201 is detected includes 140, via the sensing unit 140 to sense the and it may include a controller 180 that displays the screen on the display unit 151.”, and “[0098] Controller 180 if more than the sensed tension and the reference value is set by comparing the detection result of the tension sensor group (143-1, 143-2), the reference value is the tension sensor detects a finger being moved may be determined. Thus, on the basis of the data stored in the memory 170, in accordance with the detection result of the first tension sensor (143-1) and the second tension sensor (143-2), to unlock the screen of the display section 151 or, It can display and select a menu, and the user may execute the selected application.”, see also FIGs. 3-16, “[0020] Herein from the description that the electronic apparatus include wearable devices (wearable device, for example,... glass-type terminals (smart glass), HMD (head mounted display)), etc. should be included can is.”, [0043], [0083], and [0092]);
the hand of the user wearing the controller apparatus (Lee: FIG. 2);
detecting a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the controller apparatus (Lee: FIGs. 2-3 and “[0090] Sensing unit 140 detects a tension change in the wearable unit 201 provides the detection result to the controller 180. Here, the tension (tension) may refer to the force applied to the line. Wearing the wearable unit 201 and the finger piece state can be measured by the tensile force is zero, when the fingers are folded to the tension generated in the wearable unit 201. Thus, the sensing unit 140 may provide the sensed value to the controller 180 when the tension of the wearable unit 201 changes. The sensing unit 140 may be provided in a region easy to detect tension changes in accordance with the finger motion.”); and
displaying an interaction-related operation on an image displayed on the display device in response to receiving operation detection data corresponding to the detected change in the extension or contraction of the wire (Lee: FIG. 7, “[0111] If the user does not take any action in a state wearing the wearable unit 201 in the hand, that is, when the input has not occurred the controller 180 may be a display unit 151 may display a lock (LOCK) screen . Thus, when the user fold the index finger and ring finger at the same time the first tension value (TS_1) and the second tension value (TS_2) is input at the same time as the controller 180.”, and “[0112] The controller 180 may display a first tension value (TS_1) and the second tension value if found to be (TS_2) is greater than the reference value is pre-set, the secondary display a user input standby state in the standby screen 151.”, see also FIG. 1 (haptic module 153), FIGs. 8-16 and [0067]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Lee, such that Motta as modified teaches: An operating method of an XR device, the method comprising (operating method and XR device of Motta as modified): connecting the XR device and a controller apparatus worn on a user's hand (connecting of Motta as modified combined with connecting of Lee); recognizing the hand of the user wearing the controller apparatus through a camera of the XR device (recognizing of Motta as modified combined with the hand of the user wearing the controller apparatus of Lee); detecting a change in extension or contraction of a connected wire according to a motion of the hand of the user wearing the controller apparatus (detecting of Motta as modified combined with detecting of Lee); and displaying an interaction-related operation on a VR image displayed on the XR device in response to receiving operation detection data corresponding to the detected change in the extension or contraction of the wire (displaying of Motta as modified combined with displaying of Lee), for user interaction with a VR image using a controller apparatus.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Lee, in further view of Bae et al. in KR 10-1628703 B1 (hereinafter Bae; January 7, 2025 IDS reference – an original copy and a translation of the abstract was submitted with the IDS; an original copy and a full machine translation thereof is/was provided with the first Office action mailed in response to the filing of the instant application and preliminary amendment), and in further view of Kearney et al. in US 2018/0284896 A1 (hereinafter Kearney).
Regarding claim 8, Motta as modified by Lee teaches:
The controller apparatus of claim 7.
However, it is noted that Motta as modified by Lee does not teach:
wherein the sensor detects, subsequent to wearing a ring, which is worn in a fitted form on the user's finger, while the ring and a wire wrapped around a wheel module are connected, wire extension corresponding to a motion of bending the finger or wire contraction corresponding to a motion of extending the finger, and transmits the detected wire extension or contraction to the processor.
Bae teaches:
wherein a sensor (e.g., 131 in FIG. 4) detects, subsequent to wearing a ring (e.g., 111 in FIGs. 3-4), which is worn in a fitted form on a user's finger (see FIG. 3), while the ring and a wire (e.g., 121 in FIGs. 3-4) are connected, wire extension corresponding to a motion of bending the finger or wire contraction corresponding to a motion of extending the finger, and transmits the detected wire extension or contraction (Bae: FIGs. 3-4, p. 4, ¶ 1 (“... [T]he ring-shaped structure [110], which can be worn on a human finger in ring form and can be adjusted in size to fit fingers of various sizes. consists of a first ring-shaped structure [111] fitted into the second node and a second ring-shaped structure [112] fitted into the third joint.”), ¶ 2 (“The carbon wire [120]... consists of first and second carbon wires [121, 122] that are respectively connected to the first and second ring structures [111, 112].”), ¶ 3 (“... [T]he carbon wire [120] has its own elasticity,... and by using a very thin carbon rod, finger movement can be measured without affecting finger movement.”), ¶ 5 (“The sensing module [130] consists of first and second potentiometers [131, 132] each connected to the other ends of the first and second carbon wires [121, 122].”), and ¶ 12 (“... [A] finger movement measurement system using linear potentiometers [131, 132] and the carbon wire [120] is proposed. The carbon wire [120] is attached to the back of the finger. By finger movement, the joint angle can be calculated by measuring the change in length of the carbon wire [120] as the carbon wire [120] moves.”), see also p. 4, 2nd to the last ¶ and last ¶ (“... As the fingers bend, the creases of the finger joints stretch, causing the connected lines to shift. The shifted distance ΔL is calculated....“), p. 5, ¶ 2 (“... Length change ΔL. As shown in Figure 5c, it is measured by a linear potentiometer installed as ΔP....”) and ¶ 5 (“When the finger is extended to its original position, the carbon wire [120] returns to its original position by its own elasticity.... [T]he carbon wire [120] transmits finger movement to the potentiometer and maintains its original shape by its own elasticity.”), and p. 6, ¶ 4).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Bae, to detect finger motion input.
However, it is noted that Motta as modified by Lee and Bae does not teach:
the wire wrapped around a wheel module.
Kearney teaches:
a wire (tensile element) wrapped around a wheel module (pulley) (Kearney: FIGs. 3-4, “[0098] As shown in FIG. 3, the device comprises a plurality of tensile element configured to couple different portion of each finger of the user, specifically, a distal tensile element 402 configured to couple to a fingertip segment of the finger using fingertip pad 401 and a proximal tensile element 404 configured to couple to a proximal segment of a finger using proximal pad 403. As further described herein, the tensile elements are in operable communication with and cooperate with both the position detection module and motor control module to: produce positional data of the various parts of the finger, produce a predetermined range of motion for coupled portions of the finger, produce applied force to the coupled portions of the finger as part of the force feedback.”, and “[0099] As shown in FIG. 4, each position detection module 104 comprises a pair of cylinder elements containing positional sensors, specifically a base segment pulley (410a, 410b, 410c, 410d, 410e) with positional sensor and a middle segment pulley (411a, 411b, 411c, 411d, 411e) with positional sensor. Referring again to FIG. 3, the tensile elements, which may comprise a filament or string, are connected to and configured to be pulled from and wound up into the pulley. The pulleys comprise a retraction mechanism, such as a torsion spring, to maintain a tension level and to return the tensile element to a start or default position. The positional data is determined based at least on measuring tensile element stroke length caused by movement of the finger. In operation, the sensors measure rotational motion data from the cylinder element for a plurality of axes, and linear displacement of the tensile element is determined by mapping rotational data to a linear distance based on cylinder dimensions. In further aspects, the force exerted by the retraction system to maintain the tension level and or retract the tensile element is minimal such that it cannot be perceived by the user or effect the force feedback effects.”).
Before the effective filing date of the claimed invention, it would have been obvious to include: the features taught by Kearney, such that Motta as modified teaches: wherein the sensor detects, subsequent to wearing a ring, which is worn in a fitted form on the user's finger, while the ring and a wire wrapped around a wheel module are connected, wire extension corresponding to a motion of bending the finger or wire contraction corresponding to a motion of extending the finger, and transmits the detected wire extension or contraction to the processor (sensor and processor of Motta as modified combined with the sensor detects of Bae and the wire wrapped around a wheel module of Kearney; also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to detect finger motion input), to detect finger motion input as taught by Kearney.
Regarding claim 9, Motta as modified by Lee, Bae, and Kearney teaches:
The controller apparatus of claim 8, wherein the controller apparatus has a structure in which one end of the wire is connected to the ring, and the other end of the wire is wound in the form of a fixed cable around a torsion spring built into the wheel module (Bae: one end of the wire is connected to the ring, and the other end of the wire is connected to the sensor; FIGs. 3-4 and p. 4, ¶ 2 (“The carbon wire [120]... consists of first and second carbon wires [121, 122] that are respectively connected to the first and second ring structures [111, 112].”) and ¶ 5 (“The sensing module [130] consists of first and second potentiometers [131, 132] each connected to the other ends of the first and second carbon wires [121, 122].”); Kearney: one end of the wire is connected to a finger pad, and the other end of the wire is wound as claimed; FIGs. 3-4, “[0098] As shown in FIG. 3, the device comprises a plurality of tensile element configured to couple different portion of each finger of the user, specifically, a distal tensile element 402 configured to couple to a fingertip segment of the finger using fingertip pad 401 and a proximal tensile element 404 configured to couple to a proximal segment of a finger using proximal pad 403. As further described herein, the tensile elements are in operable communication with and cooperate with both the position detection module and motor control module to: produce positional data of the various parts of the finger, produce a predetermined range of motion for coupled portions of the finger, produce applied force to the coupled portions of the finger as part of the force feedback.”, and “[0099] As shown in FIG. 4, each position detection module 104 comprises a pair of cylinder elements containing positional sensors, specifically a base segment pulley (410a, 410b, 410c, 410d, 410e) with positional sensor and a middle segment pulley (411a, 411b, 411c, 411d, 411e) with positional sensor. Referring again to FIG. 3, the tensile elements, which may comprise a filament or string, are connected to and configured to be pulled from and wound up into the pulley. The pulleys comprise a retraction mechanism, such as a torsion spring, to maintain a tension level and to return the tensile element to a start or default position. The positional data is determined based at least on measuring tensile element stroke length caused by movement of the finger. In operation, the sensors measure rotational motion data from the cylinder element for a plurality of axes, and linear displacement of the tensile element is determined by mapping rotational data to a linear distance based on cylinder dimensions. In further aspects, the force exerted by the retraction system to maintain the tension level and or retract the tensile element is minimal such that it cannot be perceived by the user or effect the force feedback effects.”; claim 7 above (controller apparatus); also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to detect finger motion input), and
wherein the sensor operates to detect wire extension when the wire is withdrawn in a direction of the ring according to a motion of bending the finger, and detect wire contraction when the wire is withdrawn in a direction of the wheel module according to a motion of extending the finger (Bae: the sensor operates to detect wire extension as claimed, and detect wire contraction when the wire is withdrawn in a direction of the sensor according to a motion of extending the finger; FIGs. 3-4, p. 4, ¶ 2 (“The carbon wire [120]... consists of first and second carbon wires [121, 122] that are respectively connected to the first and second ring structures [111, 112].”), ¶ 3 (“... [T]he carbon wire 11201 has its own elasticity,... and by using a very thin carbon rod, finger movement can be measured without affecting finger movement.”), ¶ 5 (“The sensing module [130] consists of first and second potentiometers [131, 132] each connected to the other ends of the first and second carbon wires [121, 122].”), and ¶ 12 (“... [A] finger movement measurement system using linear potentiometers [131, 132] and the carbon wire [120] is proposed. The carbon wire [120) is attached to the back of the finger. By finger movement, the joint angle can be calculated by measuring the change in length of the carbon wire [120] as the carbon wire [120] moves.”), see also p. 4, 2nd to the last ¶ and last ¶ (“... As the fingers bend, the creases of the finger joints stretch, causing the connected lines to shift. The shifted distance ΔL is calculated....“), p. 5, ¶ 2 (“... Length change ΔL. As shown in Figure 5c, it is measured by a linear potentiometer installed as ΔP....”) and ¶ 5 (“When the finger is extended to its original position, the carbon wire [120] returns to its original position by its own elasticity.... [T]he carbon wire [120] transmits finger movement to the potentiometer and maintains its original shape by its own elasticity.”), and p. 6, ¶ 4; Kearney: the sensor operates to detect wire extension when the wire is withdrawn in a direction of the finger pad according to a motion of bending the finger, and detect wire contraction as claimed; FIGs. 3-4 and [0098]-[0099]; also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to detect finger motion input).
Claims 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Motta in view of Lee, in further view of Gonzalez Franco.
Regarding claim 10, Motta as modified by Lee teaches:
The controller apparatus of claim 7, wherein the processor outputs vibration feedback through the output module, and an operation of interacting with a target object by a virtual object of the VR image is displayed based on the operation detection data (Lee: FIGs. 1-3, “[0028] Output unit 150 is for generating an output related to visual, auditory or tactile sense, a display unit 151, the audio output unit 152, haeptip module 153, a light output portion 154 of the at least one can do....”, “[0031] The controller 180 is in addition to the operation associated with the application, typically controls the overall operations of the electronic device 100. Controller 180 by driving the application stored in the process a signal, data, information, such as input or output, or the memory 170 through the components shown above, allows the user to provide or process the appropriate information or features.”, “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C] (e.g., IR cameras with IR illumination) that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0045]... HMD 2000 [in FIG. 4] may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection.... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”, “[0067] The haptic module (haptic module) (153) generates a variety of tactile effects which a user can feel. The haptic module 153, a typical example of the tactile effects that occur may be a vibration....”, “[0086] Electronic device 200 related to the embodiment of the present invention may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”, and “[0092] The controller 180 may be provided in a form embedded in the inside of the back plate of the wearable unit of the display unit 151 (201). Controller 180 to the sensing unit display portion 151 determines the movement of the finger, and processes the function is set to the motion of the determined finger based on the change in the tension of the wearable unit 201 is detected through a 140 It may display a screen.”).
However, it is noted that Motta as modified by Lee does not explicitly teach:
the vibration feedback is output while the operation is displayed, and
the operation is an operation of gripping the target object by the virtual object.
Gonzalez Franco teaches:
wherein a processor (of 102 in FIG. 4) outputs vibration feedback through an output module (306) while an operation of gripping a target object (602 in FIG. 6A) by a virtual object (604) of a VR image is displayed based on operation detection data (Gonzalez Franco: FIGs. 1, 2B, 4A, and 5-6A, “[0020]... The virtual reality system 100 [in FIG. 1] may include a base station 102. The base station 102 can include hardware and/or software for generating and executing a virtual reality world, including receiving and processing inputs from a user 104, and generating and outputting feedback to the user 104. The base station 102 may be any computing device, including a personal computer (PC), server, gaming console..., simulator, etc.”, “[0062]... [V]isual information that simulates the virtual fingers 604 moving in the direction of the arrows 606 shown in FIG. 6A and grasping the virtual object 602 [is generated]. This visual information may be transmitted... to the display 514 and rendered to the user 104....”, and “[0063]... [T]he base station 102 may... generate haptic information that simulates the virtual fingers 604 grasping the virtual object 602....For example, in a scenario where the virtual object 602 is a potato chip, if the user's fingers 206 apply gentle forces on the finger rests 208 when handling the virtual potato chip, then... little to no haptic feedback [is generated]. But if the user's fingers 206 squeeze... too hard, then... the virtual potato chip cracking or breaking, and generate high-frequency vibrations as haptic feedback, thus rendering a crisp sensation to the user's fingers 206 [may be simulated]. The haptic information may be transmitted... to the controller 108 and rendered to the user's fingers 206 via the actuators 306....”, see also FIG. 4B).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include: the features taught by Gonzalez Franco, such that Motta as modified teaches: wherein the processor outputs vibration feedback through the output module while an operation of gripping a target object by a virtual object of the VR image is displayed based on the operation detection data (processor, vibration feedback, output module, operation, target and virtual objects, VR image, and operation detection data of Motta as modified combined with the processor outputs of Gonzalez Franco), to provide a realistic user experience. (Gonzalez Franco: “[0063]... [T]he combination of contemporaneous visual feedback and haptic feedback may create compelling realistic experience for the user 104....”).
Regarding claim 11, Motta as modified by Lee and Gonzalez Franco teaches:
The controller apparatus of claim 10, wherein when an increase in wire extension is detected by the sensor while an operation of gripping a target object by a virtual object of the VR image is displayed, a level of vibration feedback output through the output module is increased to transmit data related to the increase in wire extension to the XR device so as to display a feedback image around the target object of the VR image (Motta: “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C]... that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection..... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”; Lee: FIG. 1, “[0028] Output unit 150 is for generating an output related to visual, auditory or tactile sense, a display unit 151, the audio output unit 152, haeptip module 153, a light output portion 154 of the at least one can do....”, “[0067]... The haptic module 153 [in FIG. 1], a typical example of the tactile effects that occur may be a vibration. The haptic module 153 is vibration intensity and pattern of generated from such may be controlled by a set of user choice or control. For instance, the haptic module 153 may output or sequentially output different vibration to each other synthesized.”, and “[0086] Electronic device 200... may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”; Gonzalez Franco: FIG. 6A, “[0062] The base station 102 may generate visual information that simulates the virtual fingers 604 moving in the direction of the arrows 606 shown in FIG. 6A and grasping the virtual object 602....”, and “[0063]... [T]he base station 102 may... generate haptic information that simulates the virtual fingers 604 grasping the virtual object 602.... [T]he characteristics of the haptic information, including amplitude, frequency, and/or duration, may depend on several factors, such as the level of force applied by the fingers 206 on the finger rests 208 and the virtual size, shape, weight, texture, structure, material, and/or orientation of the virtual object 602. For example, in a scenario where the virtual object 602 is a potato chip, if the user's fingers 206 apply gentle forces on the finger rests 208 when handling the virtual potato chip, then... little to no haptic feedback [is generated]. But if the user's fingers 206 squeeze... too hard, then... the virtual potato chip cracking or breaking, and generate high-frequency vibrations as haptic feedback, thus rendering a crisp sensation to the user's fingers 206 [may be simulated]....”, see also FIGs. 6B-E; claims 7 and 10 above (level of vibration feedback increase of Gonzalez Franco and sensing gripping of Lee); it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to provide a feedback image corresponding to an increased gripping operation).
Regarding claim 12, Motta as modified by Lee and Gonzalez Franco teaches:
The controller apparatus of claim 10, wherein when wire contraction is detected by the sensor while an operation of gripping a target object by a virtual object of the VR image is displayed, a level of vibration feedback output through the output module is decreased to transmit data related to the wire contraction to the XR device so as to display a feedback movement for the target object of the VR image (Motta: “[0039]... [T]he user sensors may include one or more hand sensors 217 [in FIGs. 2A-C]... that track position, movement, and gestures of the user's hands, fingers, and/or arms. For example,... detected position, movement, and gestures of the user's hands, fingers, and/or arms may be used to simulate movement of the hands, fingers, and/or arms of an avatar of the user 290 in the virtual space. As another example, the user's detected hand and finger gestures may be used to determine interactions of the user with virtual content in the virtual space, including but not limited to gestures that manipulate virtual objects, gestures that interact with virtual user interface elements displayed in the virtual space, etc.”, “[0045]... HMD 2000 may... include one or more interfaces 2040 (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device 2100 via a wired or wireless connection..... [A]t least a part of the functionality described for the controller 2030 may be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a... game controller....”; Lee: FIG. 1, “[0028] Output unit 150 is for generating an output related to visual, auditory or tactile sense, a display unit 151, the audio output unit 152, haeptip module 153, a light output portion 154 of the at least one can do....”, “[0067]... The haptic module 153 [in FIG. 1], a typical example of the tactile effects that occur may be a vibration. The haptic module 153 is vibration intensity and pattern of generated from such may be controlled by a set of user choice or control. For instance, the haptic module 153 may output or sequentially output different vibration to each other synthesized.”, and “[0086] Electronic device 200... may optionally contain a configuration of the control block of Fig. 1, each configuration may be provided in the wearable unit 201 of the glove form that enables the user to wear.”; Gonzalez Franco: FIGs. 6A-B, “[0062] The base station 102 may generate visual information that simulates the virtual fingers 604 moving in the direction of the arrows 606 shown in FIG. 6A and grasping the virtual object 602....”, “[0063]... [T]he base station 102 may... generate haptic information that simulates the virtual fingers 604 grasping the virtual object 602.... [T]he characteristics of the haptic information, including amplitude, frequency, and/or duration, may depend on several factors, such as the level of force applied by the fingers 206 on the finger rests 208 and the virtual size, shape, weight, texture, structure, material, and/or orientation of the virtual object 602. For example, in a scenario where the virtual object 602 is a potato chip, if the user's fingers 206 apply gentle forces on the finger rests 208 when handling the virtual potato chip, then... little to no haptic feedback [is generated]. But if the user's fingers 206 squeeze... too hard, then... the virtual potato chip cracking or breaking, and generate high-frequency vibrations as haptic feedback, thus rendering a crisp sensation to the user's fingers 206 [may be simulated]....”, and “[0065] The base station 102 may generate visual information that simulates the virtual fingers 604 moving away from the virtual object 602 in the direction of the arrows 608 shown in FIG. 6B and releasing the virtual object 602. If the virtual object 602 is not already resting on a virtual surface, the base station 102 may simulate the virtual object 602 falling, spinning, hitting a virtual surface, tumbling, etc., according to the virtual world scenario.... Realistic experience may be provided to the user 104, who pulled her fingers 206 off the finger rests 208, by contemporaneously showing the virtual fingers 604 opening and thereby releasing the virtual object 602.”, see also FIGs. 6C-E; claims 7 and 10 above (i.e., level of vibration feedback decrease of Gonzalez Franco and sensing ungripping of Lee); also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to provide a feedback image corresponding to a decreased gripping operation).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to K. Kiyabu whose telephone number is (571) 270-7836. The examiner can normally be reached Monday to Thursday 9:00 A.M. - 5:00 P.M. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Temesghen Ghebretinsae, can be reached at (571) 272-3017. The fax number for the organization where this application or proceeding is assigned is (571) 273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicants are encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/patents/uspto-automated-interview-request-air-form.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/K. K./
Examiner, Art Unit 2626
/TEMESGHEN GHEBRETINSAE/Supervisory Patent Examiner, Art Unit 2626 6/22/26B
Prosecution Insights