Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Applicant’s amendments filed on 07/14/2026 have been received and considered. Claims 1-20 are pending. No new claims have been amended, added, or cancelled.
Response to Arguments
Applicant’s arguments, see pgs. 8-10, filed 07/14/2026, with respect to the rejections of claims 1 and 15 under 35 U.S.C. 103 have been fully considered and are persuasive.
More specifically, the Applicant argues that “Tokubo’s machine learning model for training haptic feedback operates by tracking physical interactions in the real world” (Remarks, pg. 9, lines 4-5), as opposed to “digital-based ‘time series of orientation data’ and ‘positioning data with respect to the virtual object’ of the digital avatar from the XR application” (Remarks, pg. 9, lines 10-12).
The Examiner agrees. Therefore, the 103 rejections of claims 1 and 15 have been withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-5, and 7-11, 15-16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zurmoehle et al. (US 2021/0335053 A1, hereinafter Zurmoehle), in view of Tokubo. (US 2023/0041294 A1), and further in view of Wu et al. (US 2018/0232051 A1, hereinafter Wu).
Regarding claim 1, Zurmoehle teaches a method comprising: ([Abstract] “Described are improved systems and methods for navigation and manipulation of interactable objects in a 3D mixed reality environment.”)
generating for display a virtual object, within an extended reality (XR) environment, ([0009] “The improved systems and methods for navigation and manipulation of browser windows may be applied in the context of 2D content that is deconstructed and displayed in a spatially organized 3D environment. This may include identifying 2D content, identifying elements in the 2D content, identifying surrounding surfaces, mapping the identified elements to the identified surrounding surfaces, and displaying the elements as virtual content onto the surrounding surfaces.”)
wherein the display is based on at least one generic user interface (UI) element, ([0097] “In some embodiments, the virtual objects may be extracted objects, wherein an extracted object may be a physical object identified within the user's physical environment 105… Additionally, extracted objects may be virtual objects extracted from the 2D content (e.g., a web page from a browser) and displayed to the user 108. For example, a user 108 may choose an object such as a couch from a web page displayed on a 2D content/web page to be displayed within the user's physical environment 105. The system may recognize the chosen object (e.g., the couch) and display the extracted object (e.g., the couch) to the user 108 as if the extracted object (e.g., the couch) is physically present in the user's physical environment 105.”)
wherein design of the XR environment is based at least in part on: providing for display, at a first device, a design user interface for creating the XR environment; ([0069] “As an example, 2D content accessed or displayed by the web browser 110 may be a web page having multiple tabs, wherein a current active tab 260 is displayed and a secondary tab 250 is currently hidden until selected upon to display on the web browser 110… Additionally, the secondary tab 250 may be mapped to display on a virtual Rolodex 190 and/or on a multi-stack virtual object 194.”, where “FIG. 2 illustrates an example mapping of elements of a 2D content to a user's 3D environment, according to some embodiments. Environment 200 depicts a 2D content (e.g., a web page) displayed or accessed by a web browser 110 and a user's physical environment 105” [0068] and “Additionally, the web browser 110 may also be any technology that displays digital 2D content… In some embodiments, the web browser 110 containing 2D content (e.g., web page) is displayed via computing network 125.” [0056]. Note: the web browser 110 is mapped to a first device, the Rolodex 190 is mapped to a design user interface.)
receiving a design user interface selection for placement of the at least one generic UI element in a location in the XR environment, ([0072] “The user 108 may bi-directionally cycle through content within the virtual Rolodex 190 by simply focusing on a particular tab within the virtual Rolodex 190 and the one or more sensors (e.g., the sensors 162) within the head-mounted system 160 will detect the eye focus of the user 108 and cycle through the tabs within the virtual Rolodex 190 accordingly to obtain relevant information for the user 108. In some embodiments, the user 108 may choose the relevant information from the virtual Rolodex 190 and instruct the head-mounted system 160 to display the relevant information onto either an available surrounding surface or on yet another virtual object such as a virtual display in close proximity to the user 108”)
wherein the at least one generic UI element represents a functionality of a real-world object; ([0097] “In some embodiments, the virtual objects may be extracted objects, wherein an extracted object may be a physical object identified within the user's physical environment 105, but is displayed to the user as a virtual object in the physical object's place”)
configuring an XR application to run on a second device, wherein the XR application causes display of the XR environment; ([0123] “At 1001, one or more interactable objects are identified and presented to the user through the display device of the mixed reality system for a host application”, where “The real objects may be displayed either as digital computer generated display object renderings that corresponds to the real object (e.g., for a VR application) or viewable as the real object itself through a transparent portion of the display device (e.g., for an AR application).” And “The virtual objects are objects that only exist as constructs within the virtualized universe and are generated/displayed as computer-generated renderings to the user through a head-mounted display device.” Note: the head-mounted display is mapped to the second device, and the host application, which is stated to either be an AR or VR application, is mapped to the XR application.)
configuring middleware to run on the second device, ([0142] “The head-mounted system 160 may be a virtual reality (VR) or augmented reality (AR) head-mounted system that includes a user interface, a user-sensing system, an environment sensing system, and a processor”, where “The user interface may be at least one or a combination of a haptics interface devices, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device and an audio input device.” [0143]. Note: the user interface is mapped to the middleware).
wherein the middleware is configured to control at least one haptic device ([0143] “A haptics interface device, such as totem 1102, is a device that allows a human to interact with a computer through bodily sensations and movements. Haptics refers to a type of human-computer interaction technology that encompasses tactile feedback or other bodily sensations to perform actions or processes on a computing device.”)
based on detecting movement of an avatar in the XR environment by the XR application in a vicinity of the virtual object, ([0125] “At 1005, the mixed reality system identifies and tracks a physical movement of the user with respect to the selected object. For example, while the user has selected an object at a first location, the user may desire to move that object to a second location. With the present embodiment, that movement of the object to the second location may be implemented by physical movement of the user from the first location to the second location.” Note: the avatar is mapped to the user.)
Zurmoehle fails to teach to control at least one haptic device using at least one neural network; based on detecting movement of an avatar in the XR environment by the XR application in a vicinity of the virtual object, causing the middleware to put input data into the at least one neural network, wherein the input data comprises: (i) a time series of user pose data received from at least one sensor, (ii) a time series of orientation data of the avatar received from the XR application, and (iii) positioning data of the avatar with respect to the virtual object received from the XR application, wherein the at least one neural network of the middleware outputs control data for controlling the at least one haptic device based at least in part on the input data; and controlling, by the middleware, the at least one haptic device based on the control data. However, it is known in the art as taught by Tokubo.
Tokubo teaches wherein the middleware is configured to control at least one haptic device using at least one neural network; ([0044] “In thus using the above, an ML model can be trained for dynamic, on-the-fly haptic feedback generation over time along various points on the object itself”, where “Also note here that the haptic feedbacks themselves may be generated using various vibration generators positioned at various points within the object itself.” [0046])
wherein the input data comprises: (i) a time series of user pose data received from at least one sensor ([0044] “In thus using the above, an ML model can be trained for dynamic, on-the-fly haptic feedback generation over time along various points on the object itself, depending on hand pose”, where “Commencing at block 600, the hand is imaged and the pose identified at block 602 using a camera and image recognition/CV techniques (and/or using the other sensors described above).” [0037])
(ii) a time series of orientation data ([0044] “In thus using the above, an ML model can be trained for dynamic, on-the-fly haptic feedback generation over time along various points on the object itself, depending on… known contact points/grip of the hand against various locations of the object”, where “Additionally, grip pose and object pose may also be used to distinguish fine motor interactions with virtual objects in a simulation from gross motor interactions based on how the corresponding real-world object is held and in which orientation to assist the device in determining which type of motor interaction is being performed.” [0050]), and “note that the simulation may be a computer game or other three-dimensional or VR simulation, for example.” [0046])
(iii) positioning data of the user with respect to the ([0044] “In thus using the above, an ML model can be trained for dynamic, on-the-fly haptic feedback generation over time along various points on the object itself, depending on… known contact points/grip of the hand against various locations of the object”, where “Note that information about the position, orientation, and type of object being held may also be used to correct hand tracking” [0049]. Note: the “contact points/grips of the hand against various locations of the object” is mapped to positioning data.)
wherein the at least one neural network of the middleware outputs control data for controlling the at least one haptic device based at least in part on the input data; ([0045] “Also, other haptics for other poses/hand grips (but possibly for the same virtual action) may then be inferred using this preprogramming and the trained ML model itself.”)
and controlling, by the middleware, the at least one haptic device based on the control data ([0046] “Also note here that the haptic feedbacks themselves may be generated using various vibration generators positioned at various points within the object itself… The haptics produced by the vibration generators may therefore mimic similar vibrations/forces for the corresponding virtual element of the simulation itself that is represented by the real-world object. Again note that the simulation may be a computer game or other three-dimensional or VR simulation, for example.”)
Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that “Thus, haptic feedbacks for the same computer simulation effect may be rendered differently according to the actual contact points of the hand, hand pose, and the object's own pose so that the rendered haptics vary based on whether the object is held, e.g., in the palm, or open-handed, or with only fingers, etc.” [0045]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to Zurmoehle to input user pose data, user orientation data, and user positioning data in respect to the virtual object into the machine learning model in order to account the haptic feedback for the various ways in which the object can be held by the user.
The combination of Zurmoehle and Tokubo fails to teach orientation data of the avatar received from the XR application, and (iii) positioning data of the avatar user with respect to the virtual object received from the XR application. However, this is known in the art as taught by Wu.
Wu teaches orientation data of the avatar received from the XR application ([Abstract] “the dynamic generation of the haptic effects are based on the locations of the first and second haptic output devices, the position and orientation of the user's avatar in relationship to the position and the type of video event.”)
(iii) positioning data of the avatar user with respect to the virtual object received from the XR application ([0013] “Based on the avatar's position, the position of the multiple haptic output devices, the type of detected video event, and the position and orientation to the video event, dynamic localized haptics are generated for each haptic output device.”, where “In addition, an event is not limited to an explosion or collision but rather be any specific interaction the user has with the virtual environment, including something simply touching an object.” [0047]).
Wu is analogous to the claimed invention, as both relates to haptic feedback based on avatar interaction with a virtual environment. Wu further teaches “In contrast to the known systems, embodiments in the current disclosure allow users to experience haptic feedback added to their user experience so that the users can experience a more realistic surrounded based tactile environment based on their avatar's position in the virtual environment, haptic output device placement on the user, event type and event position.” [0028]. Therefore, it would be obvious for one of ordinary skill in the art to incorporate the teachings of Wu to the combination of Zurmoehle and Tokubo to have a more realistic experience in a virtual environment.
Regarding claim 2, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, further comprising: pre-training the at least one neural network for generating the control data based on at least one functional description of the at least one generic UI element (Tokubo; [0041] “Object types may also be included in the training set in certain examples so that when the ML model executes the logic of FIG. 6 it may also account for object type in selecting the haptic feedback”, where “FIG. 2, haptic feedback may be generated on the object being held to mimic the tactile sensation of the object in hand (e.g., haptic feedback along the length and circumference of portions of the elongated object identified as being held, but no haptic feedback at other object locations). Haptic feedback that can be correlated to a hand pose and, if desired, an object type includes intermittent buzzing, continuous shaking, isolated bumps.” [0033]).
Similarly to claim 1, Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that “it may also account for object type… so that, for example, harder or denser objects generate higher-intensity haptic feedback than softer or less dense objects.” [0041]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to the combination of Zurmoehle, Tokubo, and Wu to generate control data based on functional description in order to account for the properties that the real-world equivalent of the virtual object in regards to haptic feedback.
In regards to claim 4, the combination of Zurmoehle, Tokubo, and Wu further teaches the method of claim 1, wherein the generating for display the virtual object further comprises: determining, by the middleware, at least one of: (a) a type of sensor, or (b) a type of the at least one haptic device; (Tokubo; [0034] “further note that hand poses and particular hand contact points along the object 204 may also be determined using various other sensors in addition to or in lieu of a camera, in any appropriate combination.”)
transmitting, to the XR application, data comprising the at least one of: (a) the type of sensor, or (b) the type of the at least one haptic device; (Tokubo; [0037] “FIG. 6 further illustrates present principles. Commencing at block 600, the hand is imaged and the pose identified at block 602 using a camera and image recognition/CV techniques (and/or using the other sensors described above)”, where “Known object physics for haptic feedback for a given object… may therefore be applied differently for a given computer simulation effect depending on which hand pose/object pose combination is being used, which points of the object are contacted by the person's hand, and/or the desired effect itself according to whatever is being simulated haptically as part of the computer simulation.” [0044]).
and wherein appearance of the virtual object is selected by the XR application based at least in part on the at least one of: (a) the type of sensor, or (b) the type of the at least one haptic device (Tokubo; [0034] “For example, pressure sensors and capacitive or resistive touch sensors located at various points along the exterior of the object's housing may be used to determine hand pose/contact points. Ultrasound transceivers within the object 204 may also be used to survey the surface of the object 204 for determining hand pose/contact points, as may strain sensors to identify where the object's housing is warping to thus infer contact points at warp points.”).
Similarly to claim 1, Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that the embodiment “relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements” [0001]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to the combination of Zurmoehle, Tokubo, and Wu for improvements of haptic feedback technology in AR application.
Regarding claim 5, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 4, further comprising: causing the middleware to input, into the at least one neural network, the data comprising the at least one of: (a) the type of sensor, or (b) the type of the at least one haptic device, (Tokubo; [0044] “an ML model can be trained for dynamic, on-the-fly haptic feedback generation over time along various points on the object itself, depending on hand pose, known contact points/grip of the hand against various locations of the object, and/or the object's own pose/orientation”, where [0034] “further note that hand poses and particular hand contact points along the object 204 may also be determined using various other sensors in addition to or in lieu of a camera, in any appropriate combination.”)
wherein the control data output by the at least one neural network is based on the data comprising the at least one of: (a) the type of sensor, or (b) the type of the at least one haptic device (Tokubo; [0044] “Known object physics for haptic feedback for a given object, as preprogrammed by a developer or provided by the computer simulation itself, may therefore be applied differently for a given computer simulation effect depending on which hand pose/object pose combination is being used, which points of the object are contacted by the person's hand, and/or the desired effect itself according to whatever is being simulated haptically as part of the computer simulation.”).
Similarly to claim 1, Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that the embodiment “relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements” [0001]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to the combination of Zurmoehle, Tokubo, and Wu for improvements of haptic feedback technology in AR application.
Regarding claim 7, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, wherein the detecting the movement of the avatar in the XR environment by the XR application in the vicinity of the virtual object further comprises: detecting movement, by the at least one sensor, of at least one component of the at least one haptic device; (Zurmoehle; [0125] “The physical movement of the user may be recognized and tracked by identifying any changes of position for a controller device that is being held by the user, e.g., by tracking the physical movement of a haptics controller held by the user via positional changes of any sensors/emitters that may be embedded within the controller device and/or visually tracking movement of any markers on the controller device.”)
and generating the movement of the avatar, by the XR application, wherein the movement of the avatar corresponds to the movement of the at least one component of the at least one haptic device. (Zurmoehle; [0138] “The physical movements may be translated into event objects generated by and/or pertaining to the operation of the totem device, which correlates to data for the movement and/or positioning of totem (1058)”, where “The user input device may be a haptics controller. The haptics controller may correspond to a totem device having at least six degrees of freedom. The physical movement of the user may be translated into event objects at a processing system associated with the haptics controller, where the event objects correlate to data indicating movement or positioning of the haptics controller” [0022]).
Regarding claim 8, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, wherein the detecting the movement of the avatar in the XR environment by the XR application in the vicinity of the virtual object further comprises: receiving a user interface interaction, via a control of the second device, associated with the avatar; (Zurmoehle; [0012] “In one embodiment, a method includes receiving data indicating a selection of an interactable object contained within a first prism at the start of a user interaction. The method also includes receiving data indicating an end of the user interaction with the interactable object. The method further includes receiving data indicating a physical movement of the user corresponding to removing the interactable object from the first prism between the start and the end of the user interaction.”, where “Alternatively, selection may occur with only physical movements/gestures by the user—without the need for a user to manipulate a control device. For example, the mixed reality system may employ one or more camera devices that track the movements of a user's hand/arm, and can therefore identify when the user's hand is hovering over an interactable object/link.” [0124])
and generating the movement of the avatar, by the XR application, based on the user interface interaction (Zurmoehle; [0126] “While the user is engaged in the physical movement, at 1007, the data associated with the interactable object(s) are correspondingly displayed into new positions/locations that correlate to the user's movements. For example, if the user has selected an object with a selection arm and is currently moving the selection arm from a first location to a new location, the data associated with the interactable object (i.e. selected object) will be visually displayed in the second location that corresponds to the movement of the user's arm.”)
Regarding claim 9, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, wherein the controlling, by the middleware, the at least one haptic device based on the control data further comprises: controlling, by the middleware, at least one actuator of the at least one haptic device, (Tokubo; [0046] “the haptic feedbacks themselves may be generated using various vibration generators positioned at various points within the object itself. Each vibration generator may include, for example, an electric motor connected to an off-center and/or off-balanced weight via the motor's rotatable shaft so that the shaft may rotate under control of the motor”)
wherein the at least one actuator generates haptic feedback in at least one component of the at least one haptic device (Tokubo; [0046] “The haptics produced by the vibration generators may therefore mimic similar vibrations/forces for the corresponding virtual element of the simulation itself that is represented by the real-world object.”)
Similarly to claim 1, Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that the embodiment “relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements” [0001]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to the combination of Zurmoehle, Tokubo, and Wu for improvements of haptic feedback technology in AR application.
Regarding claim 10, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, wherein the generating for display the virtual object further comprises: determining, by the XR application, at least one aesthetic element of the XR environment, (Zurmoehle; [0097] “For example, a user 108 may choose an object such as a couch from a web page displayed on a 2D content/web page to be displayed within the user's physical environment 105. The system may recognize the chosen object (e.g., the couch) and display the extracted object (e.g., the couch) to the user 108 as if the extracted object (e.g., the couch) is physically present in the user's physical environment 105.”)
wherein appearance of the virtual object is selected by the XR application based at least in part on the at least one aesthetic element of the XR environment (Zurmoehle; [0097] “In some embodiments, the virtual objects may be extracted objects, wherein an extracted object may be a physical object identified within the user's physical environment 105, but is displayed to the user as a virtual object in the physical object's place so that additional processing and associations can be made to the extracted object that would not be able to be done on the physical object itself (e.g., to change the color of the physical object to highlight a particular feature of the physical object, etc.).”).
Regarding claim 11, the combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, wherein the at least one sensor is at least one of a pressure sensor, temperature sensor, capacitive sensor, resistive sensor, optical camera, RGB-D camera, gyroscope, accelerometer, or flex sensor. (Tokubo; [0034] “However, further note that hand poses and particular hand contact points along the object 204 may also be determined using various other sensors in addition to or in lieu of a camera, in any appropriate combination. For example, pressure sensors and capacitive or resistive touch sensors located at various points along the exterior of the object's housing may be used to determine hand pose/contact points.” and “Likewise, various poses/orientations for the object 204 itself may be determined using other sensors within the object 204 in addition to or in lieu of using a camera. Those other sensors might include motion sensors such as gyroscopes, accelerometers, and magnetometers.” [0036]).
Similarly to claim 1, Tokubo is analogous to the claimed invention, as both relate to haptic feedback in an XR environment based on the interaction between the user and virtual object. Tokubo further teaches that the embodiment “relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements” [0001]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tokubo to the combination of Zurmoehle, Tokubo, and Wu for improvements of haptic feedback technology in AR application.
Regarding claim 15, claim 15 has substantially similar limitations to claim 1, but in a system form. The combination of Zurmoehle, Tokubo, and Wu further teaches a system comprising: (Zurmoehle; [Abstract] “Described are improved systems and methods for navigation and manipulation of interactable objects in a 3D mixed reality environment.”)
control circuity configured to: (Zurmoehle; [0303] “In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the disclosure.”).
Regarding claim 16, claim 16 has substantially similar limitations to claim 2, therefore, will be rejected under the same rationale as claim 2.
Regarding claim 18, claim 18 has substantially similar limitations to claim 4, therefore, will be rejected under the same rationale as claim 4.
Regarding claim 19, claim 19 has substantially similar limitations to claim 5, therefore, will be rejected under the same rationale as claim 5.
Claims 6 and 20 is rejected under 35 U.S.C. 103 as being unpatentable over Zurmoehle (US 2021/0335053 A1) in view of Tokubo (US 2023/0041294 A1) and Wu (US 2018/0232051 A1), and further in view of Song et al. (US 2024/0211726 A1, hereinafter Song).
The combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, but fails to teach further comprising: selecting the at least one neural network from a plurality of neural networks based on the input data received by the middleware; and downloading the at least one neural network, via the middleware, to a random-access memory (RAM) of the local device. However, this is known in the art as taught by Song.
Song teaches further comprising: selecting the at least one neural network from a plurality of neural networks based on the input data received by the middleware; and downloading the at least one neural network, via the middleware, to a random-access memory (RAM) of the local device ([0128] “The device 1000 may select at least one neural network model satisfying the neural network requirements 510 based on the neural network model information of the plurality of neural network models 500. In the embodiment illustrated in FIG. 5, because the accuracy for a dog among the recognition target objects exceeds 70% that is the minimum reference value about the accuracy of the neural network requirements and the latency is less than 250 ms that is the maximum reference value, the first neural network model 500-1 may satisfy the neural network requirements”, where “the accuracy of recognizing a dog from the input data may be 72%” [0126] and “T Referring back to FIG. 6, in operation S620, the device 1000 may register a plurality of neural network models by storing the obtained neural network model information. The device 1000 may store the obtained neural network model information in the memory 1200” [0151]).
Song is analogous to the claimed invention, as both relate to neural networks using sensor and image data. Song further teaches “A device may provide inference results about input data based on the execution environment thereof (e.g., the position or time at which the device is used) by using a neural network model suitable for the purpose of an artificial intelligence service.” Therefore, it would be obvious for one of ordinary skill of the art before the effective filing date of the claimed invention to incorporate the teaching of Song to the combination of Zurmoehle, Tokubo, and Wu in order to select and download a neural network that is most suitable for what is needed in their given use case.
Regarding claim 20, claim 20 has substantially similar limitations to claim 6, therefore, will be rejected under the same rationale as claim 6.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Zurmoehle (US 2021/0335053 A1) in view of Tokubo (US 2023/0041294 A1) and Wu (US 2018/0232051 A1), and further in view of Jonker et al. (US 12579765 B2, hereinafter Jonker).
The combination of Zurmoehle, Tokubo, and Wu teaches the method of claim 1, but fails to teach wherein the real-world object is one of a light switch, a steering wheel, or a gear stick. However, this is known in the art as taught by Jonker.
Jonker teaches wherein the real-world object is one of a light switch, a steering wheel, or a gear stick ([col. 15, lines 39-47] “For example, the user 220 may press their finger at a button element 255 location on the virtual user interface 250. The button element 255 and/or virtual user interface 250 location may or may not be overlaid on the user 220, the user's hand 230, physical objects 235, or other virtual content, e.g., correspond to a position in the physical environment, such as on a light switch or controller at which the client system 200 renders the virtual user interface button.”).
Jonker is analogous to the claimed invention, as both relate to XR application with haptic feedback, where the user is able to interact with virtual object corresponding to objects in the physical environment. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Jonker to the combination of Zurmoehle, Tokubo, and Wu, as it is known in the art of XR application with haptic feedback to account for interactions with a light switch.
Allowable Subject Matter
Claims 3, 12, and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
In regards to claim 3, the combination of Zurmoehle, Tokubo, and Wu does teach the method of claim 1, further comprising: configuring a plurality of XR applications to run on a plurality of devices (Tokubo; [0027] “In the example shown, the second CE device 50 may be configured as a computer game controller manipulated by a player or a head-mounted display (HMD) worn by a player. In the example shown, only two CE devices are shown, it being understood that fewer or greater devices may be used.”). However, the prior art taken singly or in combination do not teach or suggest the limitations of "wherein each XR application of the plurality of XR applications comprises the at least one generic UI element; receiving, from respective middleware of each device of the plurality of devices running the plurality of XR applications, user interaction data of a plurality of virtual objects corresponding to the at least one generic UI element; and re-training the at least one neural network for generating the control data based on the received user interaction data". Therefore, claim 3 is considered allowable.
In regards to claim 12, the prior art taken singly or in combination do not teach or suggest the limitations of "modifying a haptic feedback complexity of the virtual object based on a number of available sensors, wherein a greater number of available sensors corresponds to a greater haptic feedback complexity". Therefore, claim 12 is considered allowable.
In regards to claim 13, the prior art taken singly or in combination do not teach or suggest the limitations of "modifying a haptic feedback complexity of the virtual object based on a number of available actuators of the at least one haptic device, wherein a greater number of available actuators corresponds to a greater haptic feedback complexity". Therefore, claim 13 is considered allowable.
In regards to claim 17, claim 17 has substantially similar limitations to claim 3, therefore, contains allowable subject matter as explained for claim 3.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALICIA HA whose telephone number is (571)272-3601. The examiner can normally be reached Mon-Thurs 9:30 AM - 6:30 PM, and Fri 9:30 AM - 1:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALICIA HA/Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights