DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1–8 and 10–19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1–40 of U.S. Patent No. 12,294,627 B2. Although the claims at issue are not identical, they are not patentably distinct from each other for the reason set forth below.
Patented claim 1 recites an XR headset that (i) sends a signal triggering an identification output from a connected device, (ii) receives that output, (iii) determines the device's location, (iv) maps that location into a coordinate system of the XR headset, (v) updates "a device map … including a mapping of connected devices localized relative to the XR headset in the coordinate system of the XR headset," (vi) detects "a user input directed at the location," and (vii) generates "one or more commands to control the connected device in response to the user input and a determination, from the device map based on the detected user input, that the connected device is mapped to the location in the coordinate system of the XR headset." Instant claim 1 recites operations (v), (vi), and (vii) while omitting operations (i) through (iv). A claim that omits limitations of a patented claim while retaining the remainder is broader than, and obvious over, the patented claim. See MPEP 804(II)(B). Practicing patented claim 1 necessarily produces the recited device map, necessarily involves detecting user input directed at a mapped location, and necessarily involves generating a command upon the recited determination, and would therefore infringe instant claim 1. The two respects in which instant claim 1 is arguably narrower are not patentably distinct. First, "a first gesture of a first type" is a species of the patented "user input," and selecting a gesture as the form of user input is an obvious variation; see also patented claim 15, which recites detecting "a gesture by a user associated with the XR headset." Second, reciting first and second connected devices at first and second locations merely makes explicit the plurality already recited in the patented "mapping of connected devices" (plural).
Instant claim 2 is an obvious variant of patented claims 1 and 15, applying the patented operation to a second device already within the patented "mapping of connected devices." Instant claim 8 is an obvious variant of patented claim 1, the patented identification output being information received from the connected device. Instant claim 10 is an obvious variant of patented claims 1, 5, and 7, a display pattern detected by an image sensor being an image of the connected device. Instant claim 14 is substantially identical to patented claim 10, differing only in reciting "gesture" in place of "user input."
Instant claims 3, 4, 5, 6, 7, 17, and 18 are obvious variants of patented claims 15 and 16. Patented claim 15 recites detecting a gesture "indicating the location of the connected device," and, "based on the gesture," triggering "one or more identification outputs from one or more connected devices," receiving "a particular identification output...originating from the location," and determining that the connected device is located at that location. Patented claim 16 recites updating "the device map to associate the connected device with the location of the connected device indicated by the gesture." Instant claims 4 and 18 recite the same sequence — a gesture at a location, triggering an identification output from one or more connected devices, receiving the output from the device at that location, and updating the map to include that device at that location — differing only in that the triggering is expressly predicated on "a determination that no devices located at the third location … are mapped to the third location in the device map." That predicate is an obvious variant, because patented claim 15 triggers the identification outputs precisely where the gestured-toward device's location has not yet been established in the device map, which is the same condition instant claims 4 and 18 recite affirmatively; making explicit a condition already implicit in the patented claim does not render the claims patentably distinct. Instant claims 3 and 17 recite a subset of the same operations and are obvious over patented claims 15 and 16 for the same reasons. Instant claim 7 recites updating the device map based on user input, which is the gesture of patented claim 15 and the map update of patented claim 16. Instant claim 5 recites the limitation of patented claim 5, differing only in reciting the third connected device rather than the connected device — a difference of nomenclature attributable to its base claim. Instant claim 6 recites in the alternative the same three pattern-and-sensor pairings that patented claims 6, 7, and 8 recite separately, and reciting in one claim, in the alternative, what the patented claims recite in separate dependent claims is an obvious variation. The limitations instant claims 5 and 6 carry from instant claim 4 are obvious variants of patented claims 15 and 16 for the reasons set forth above.
Instant claims 11, 12, and 13 are identical in substance to patented claims 2, 3, and 4 respectively, each reciting the same triggering signal, the same retry upon a failure to receive the identification output, the same determination of that failure while the XR headset is within a threshold proximity to the device location or that location is within a field of view of an image sensor, and the same alternative stopping conditions. The only difference is the base claim from which they depend, and instant claim 1 is itself an obvious variant of patented claim 1 for the reasons set forth above.
Instant claims 15, 16, 17, 18, and 19 are method claims reciting steps corresponding to the functions of instant claims 1, 2, 3, 4, and 7/8/10 respectively, and are obvious variants of patented method claims 27 and 37, which recite the corresponding steps, for the same reasons set forth above with respect to the corresponding apparatus claims.
Claim Interpretation
The following claim interpretations apply throughout. In claim 1, "of a first type" imposes no structural or functional limitation, and "relative to the first location" requires only a spatial relationship, not pointing at, intersecting, or contacting the location. In claim 3, the recited operations bear no recited causal or temporal relationship to one another; the claim does not require the determination to be made in response to the gesture, or the update in response to the determination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 3, 7, 8, 10, 14, 15, 16, 17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1).
Regarding claim 1, Parashar discloses an extended reality (XR) headset for controlling one or more devices, the XR headset comprising: at least one memory; and at least one processor coupled to the at least one memory (Parashar discloses a head-mounted display (HMD) device 104 worn by a user 102 within a physical user space ([0019]), further detailed as HMD device 200 having see-through display panels 204, 205, a computer subsystem 206, color camera 208, depth camera 210, and a head tracking system ([0032]–[0036]). The device operates in a "mixed reality environment" combining physical and virtual elements ([0001]) and is therefore an extended reality device. Structurally, remote control device 800 comprises "one or more hardware processors 804" and "tangible computer-readable storage media (e.g., storage 806)" ([0076]); see also processor(s) 1102 and memory 1104 ([0089]). Parashar expressly recites that "the remote control device is a head mounted display device" (claim 11). The headset controls one or more devices — "remote control of a physical controlled device…using a virtual control in a mixed reality environment" ([0005]).), the at least one processor being configured to:
obtain a device map including a plurality of connected devices mapped to a plurality of locations in a physical environment relative to the XR headset in a coordinate system of the XR headset (Parashar discloses that each controlled device periodically transmits "localizable beacons (e.g., based on Bluetooth 5.1…)" ([0022]), from which "the HMD device 104 can localize the physical controlled device in the physical user space, e.g., within 10 cm accuracy in three dimensions for Bluetooth 5.1" ([0023]; see also [0068]). These locations are necessarily referenced to the headset. Localization is performed by direction-finding at the headset itself — locating operation 1004 "locates the physical controlled device...using Bluetooth 5.1 direction-finding" ([0087]), Bluetooth 5.1 providing "directional antenna support" ([0022]) — and direction-finding resolves position from angle-of-arrival and range measured at the receiving antenna, inherently yielding a position relative to that receiver. Parashar confirms as much, defining each device's position on "a view axis...between the HMD device 104 and the communications system…of the corresponding physical controlled device" ([0026]; see also [0030], [0073]). A bearing and range from the headset to the device is a location "relative to the XR headset in a coordinate system of the XR headset." The locations are maintained in a stored map: "The 3D mapping subsystem 812 maps placement…of physical objects and virtual objects within the mixed reality environment" ([0078]), and storage 1120 may store "3D maps" and "device parameters" ([0090]). Parashar further discloses persistence across encounters, distinguishing devices "registered with the HMD device 104 already" from those "not yet registered" ([0023], [0027], [0068]), consistent with "spatial registration that enables geometric persistence" ([0001]).), the device map including a first connected device mapped to a first location in the physical environment relative to the XR headset in the coordinate system of the XR headset and a second connected device mapped to a second location in the physical environment relative to the XR headset in the coordinate system of the XR headset (Parashar discloses two connected devices concurrently localized at distinct locations within a single field of view: "The light 110 and the microwave oven 112 are physical controlled devices" ([0020]), both of which "periodically transmit localizable beacons" ([0022]) and both of which are independently localized by the HMD device 104 ([0023]). Their distinct headset relative locations are reflected in the independent placement of each device's virtual control: "a virtual switch control 116 is illustrated as being displayed...in the visual proximity of the light 110. Likewise, the virtual panel control 118 is illustrated as being displayed...in the directional antenna 114 of the communication device connected to the microwave oven 112" ([0026]). Each control is placed on its own view axis between the HMD and that device's antenna ([0026], [0030]) — that is, at its own headset relative location.);
detect, from at least one user, a first gesture of a first type relative to the first location in the physical environment (Parashar discloses a "natural user interface that the user interacts with through gaze, voice, and hand gestures" ([0003]), implemented with outward-facing sensors that "may detect movements within its field of view, such as gesture-based inputs or other movements performed by a user" ([0038]). The gesture is detected relative to the first device's location because the target of the gesture is anchored there: "if the user wishes to control the light 306, she can raise a finger 310 to interact with the virtual control 300" displayed "in the proximity of the light 306" ([0048]–[0049]); and in the overlapping implementation, "she can raise a finger 410 to interact with the virtual control 400, which is positioned to overlap the bounds of the light 406" ([0055]; see also [0061], finger 510 and light 506). Parashar makes the spatial character of the gesture explicit: "activation of a virtual indicator may be inferred by co-locating the physical object (e.g., the user's finger) in the physical user space and the virtual control in the virtual user space" ([0024]), and the HMD "can detect user activation of virtual controls by tracking a user's finger relative to the generated virtual controls" ([0085]). See also [0031] (the user "can reach forward within her field of view...and 'touch' the virtual switch control 116"). Under the broadest reasonable interpretation, "of a first type" imposes no structural or functional limitation, and "relative to the first location" requires only a spatial relationship — not pointing at, intersecting, or contacting the location.); and
generate one or more commands to control the first connected device in response to the detected first gesture of the first type and a determination that the first connected device is mapped to the first location in the physical environment relative to the XR headset in the coordinate system of the XR headset (Parashar discloses generating the command in response to the gesture, and conditioning it on a spatial determination made with the aid of the stored 3D map. As to the spatial determination: "With assistance from the 3D mapping subsystem 812, the user interface controller 808 can detect interactions of physical objects and virtual objects within the user's field of view, for example, to detect user activation of a virtual control by a physical object, like a user's finger" ([0078]). As to command generation and routing to the correct device: "Each user action performed through a virtual control causes the HMD device 104 to transmit a remote control instruction … to the corresponding physical controlled device, which executes the instructed function upon receipt" ([0031]); "Responsive to detecting that the user has toggled the switch represented by the virtual control 300, the virtual control object executed by the HMD device generates and wirelessly transmits a remote control instruction to the light 306" ([0050]; see also [0056], [0074]). The "remote control engine 814 generates a remote control instruction representing the user activation of the virtual control by the user" ([0080]), transmitted "responsive to detecting the first user activation" ([0084]–[0086], operations 906, 908). Because each virtual control occupies the headset relative location of its own associated device ([0026], [0030]), the 3D mapping subsystem's determination of which virtual control the user's finger has co-located with is a determination of which connected device occupies the gestured toward location.).
However, Parashar does not expressly disclose (i) that the stored data is a device map — a record in which each of a plurality of connected devices is stored in association with its respective location, expressed in a coordinate system of the XR headset, such that a first connected device is recorded as mapped to a first location and a second connected device is recorded as mapped to a second location; or (ii) that the command is generated in response to a determination that the first connected device is mapped to the first location — a lookup against that record confirming which connected device occupies the gestured-toward location. Parashar localizes each of a plurality of connected devices in three dimensions by direction-finding performed at the headset ([0022]–[0023], [0087]), expresses those locations on a view axis between the headset and the device ([0026], [0030]), maintains a 3D mapping subsystem 812 ([0078]), and stores 3D maps and device parameters ([0090]). Parashar further generates and routes the control instruction to the "corresponding" device with the assistance of the 3D mapping subsystem ([0031], [0078]).
Regarding element (i), Giraldi's head-mounted see-through display device 102 ([0011], [0020]) establishes "a coordinate frame for the real-world environment 104" from depth data acquired by its on-board sensor "by using a simultaneous localization and mapping method" ([0012]), and localizes devices "in a coordinate frame of the augmented reality display device 102" ([0013]). It then "assigns and stores a device location for the device 108 within the coordinate frame established by the augmented reality display device 102" ([0016]; see also [0028], step 426). The record is plural and built device by device: the headset connects to "device 1 312 through device N 314" ([0022]), and "all or some steps of method 400 may be repeated for identifying any additional devices within the environment," each distinguished by manifesting "in a specified order" ([0031]). The devices include smart lights, speakers, display devices, printers, and HVAC components ([0019], [0022]).
Giraldi likewise teaches the determination clause of element (ii), consulting the stored device location to interpret a gesture at a location: "The location of the physical manifestation and/or the device in the environment also may be used as contextual information in interpreting user inputs. For example, in the case of a computer connected to multiple output devices, such as multiple smart lights, a user interaction (e.g. eye gaze location or gesture input location) indicating a location of the physical manifestation may be detected as an input for controlling a function of the computing device" ([0030]). Giraldi thus resolves which of several devices a location-indicating gesture addresses by reference to the stored locations, and displays "a virtual control panel for the device (e.g. a virtual light switch for a smart lamp)" at that location ([0030]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to record the device locations Parashar's HMD already determines ([0022]–[0023], [0026], [0087]) as Giraldi's stored device locations within the coordinate frame established by the headset ([0013], [0016], [0028]), and to condition generation of Parashar's remote control instruction on a determination from that record that the device gestured toward is the device stored at that location (Giraldi [0030]). Parashar creates the need: it distinguishes devices "registered with the HMD device 104 already" from those "not yet registered" ([0023], [0027]), presupposing a stored record of prior registrations, and warns that "some devices may enter a low power mode in which they do not continually broadcast beacons" ([0022]) — a condition in which beacon direction-finding alone cannot establish which of the plural co-located devices of [0020] the user's finger is addressing. Giraldi supplies the solution and states its benefit: localizing devices in the headset's own coordinate frame lets an augmented reality experience "be tailored to the device locations" ([0010]), and specifically resolves a gesture among "multiple smart lights" by using the stored device location "as contextual information in interpreting user inputs" ([0030]). This is the use of a known technique — a device record indexed by location in the headset's coordinate frame — to improve a similar device in the same way, yielding the predictable result of routing a gesture-initiated command to the connected device the record shows at the gestured-toward location. Both references are analogous art in the same field of endeavor — localizing and controlling networked devices from a self-localizing AR headset with outward-facing depth and image sensors ([0011]–[0013], [0020], [0032]) — addressing the same problem identified at [0004] and [0049]–[0051] of the instant specification.
Regarding claim 2, Parashar in view of Giraldi discloses the XR headset of claim 1, wherein the at least one processor is configured to: detect, from the at least one user, a second gesture of the first type relative to the second location in the physical environment; and generate one or more commands to control the second connected device in response to the detected second gesture of the first type and a determination that the second connected device is mapped to the second location in the physical environment relative to the XR headset in the coordinate system of the XR headset (Parashar applies the same reach and touch finger gesture to each of two devices at their respective locations: "In the case of the virtual switch control 116, the user 102 can reach forward within her field of view through the HMD device 104 and 'touch' the virtual switch control 116 to turn the light 110 on and off. In the case of the virtual panel control 118, the user 102 can reach forward...and 'touch' buttons on the virtual panel control 118 to...operate the microwave oven 112" ([0031]). Each control is positioned at its own device's headset relative location ([0026], [0030]), and "Each user action performed through a virtual control causes the HMD device 104 to transmit a remote control instruction...to the corresponding physical controlled device" ([0031]; see also [0078], [0080], [0084]–[0086]). Parashar does not expressly disclose that the command for the second device is generated in response to a determination that the second connected device is mapped to the second location — the same deficiency identified as element (ii) in the rejection of claim 1. Giraldi cures it in the same manner, storing a device location "within the coordinate frame established by the augmented reality display device 102" ([0016]) for each of a plurality of devices ([0022], [0031]), and applying that record to disambiguate a location-indicating gesture among several like devices: "in the case of a computer connected to multiple output devices, such as multiple smart lights, a user interaction (e.g. eye gaze location or gesture input location) indicating a location of the physical manifestation may be detected as an input for controlling a function of the computing device" ([0030]). The motivation to combine set forth in the rejection of claim 1 applies without modification; because Giraldi's record covers a plurality of devices and its [0030] teaching is directed to resolving a gesture among multiple like devices by their stored locations, application to the second connected device requires no further modification of the combined system.).
Regarding claim 3, Parashar in view of Giraldi discloses the XR headset of claim 1, wherein the at least one processor is configured to: detect, from the at least one user, a third gesture relative to a third location in the physical environment; determine that no devices are mapped to the third location in the device map; and update the device map to include a third connected device mapped to the third location in the physical environment relative to the XR headset in the coordinate system of the XR headset (For a device with which the headset has not yet paired, Parashar displays "a virtual indicator...that visually overlaps the physical controlled device" ([0023]), which the user activates by touching it, activation being "inferred by co-locating the physical object (e.g., the user's finger) in the physical user space and the virtual control in the virtual user space" ([0024], [0068]–[0069]). Parashar branches on whether the device is already in the headset's record, distinguishing one that has "not yet registered with the HMD device 104" ([0023], [0068]) from one that has "registered with the HMD device 104 already" ([0027]); upon activation the headset pairs with the previously unregistered device, having localized it "within 10 cm accuracy in three dimensions" ([0023]) and positioned its virtual control on the view axis between the HMD and that device ([0026], [0030]). Parashar does not expressly disclose that the negative determination is keyed to the location, or that the newly added device's location is stored in a coordinate system of the XR headset. Giraldi teaches both: it localizes devices "in a coordinate frame of the augmented reality display device 102" ([0013]) and "assigns and stores a device location…within the coordinate frame established by the augmented reality display device 102" ([0016], [0028]) — a location-indexed record against which a lookup either returns a device or does not — built up device by device, as "all or some steps of method 400 may be repeated for identifying any additional devices within the environment" ([0031]). Giraldi further consults that record on a gesture at a location, a "gesture input location…indicating a location of the physical manifestation" being "detected as an input for controlling a function" of the device ([0030]). The motivation to combine set forth in the rejection of claim 1 applies without modification. Giraldi additionally states the benefit of adding newly encountered devices to the location-indexed record: localizing devices in the headset's coordinate frame lets an augmented reality experience "be tailored to the device locations" ([0010]), permitting a "virtual control panel for the device (e.g. a virtual light switch for a smart lamp)" at the stored location ([0030]) — the interaction Parashar performs ([0031], [0049], [0055]), which presupposes knowing what is stored at the gestured-toward location.).
Regarding claim 7, Parashar in view of Giraldi discloses the XR headset of claim 3, wherein the at least one processor is configured to update the device map to include the third connected device mapped to the third location based on user input from the at least one user (For a device that "has not yet registered with the HMD device 104" ([0023]), Parashar predicates the addition on an affirmative user act: "Responsive to instructions by the user, such as activating (e.g., 'touching') the virtual indicator, the HMD device 104 can respond to the physical controlled device's offer by attempting to pair with the physical controlled device … Once paired, the HMD device 104 and the physical controlled device can establish a security relationship and a secure communication connection" ([0024]; see also [0069]), the device thereafter being one that "has registered with the HMD device 104 already" ([0027]). Parashar does not expressly disclose that the record so updated is a location-indexed device map in a coordinate system of the XR headset — the same deficiency identified in the rejections of claims 1 and 3. Giraldi cures it in the same manner, "assigns and stores a device location...within the coordinate frame established by the augmented reality display device 102" ([0016], [0028]) for each device as it is identified ([0031]). The motivation to combine set forth in the rejection of claims 1 and 3 applies without modification.).
Regarding claim 8, Parashar in view of Giraldi discloses the XR headset of claim 3, wherein the at least one processor is configured to update the device map to include the third connected device mapped to the third location based on information received from the third connected device (Parashar predicates localization of a previously unrecorded device on information that device transmits: "upon receiving localizable beacons from the physical controlled device, the HMD device 104 can localize the physical controlled device in the physical user space, e.g., within 10 cm accuracy in three dimensions for Bluetooth 5.1" ([0023]; see also [0068], [0087]). The device further transmits data identifying itself and its control interface prior to registration ([0029], [0071]), after which it is one that "has registered with the HMD device 104 already" ([0027]). Parashar does not expressly disclose that the record so updated is a location-indexed device map in a coordinate system of the XR headset — the same deficiency identified in the rejections of claims 1 and 3. Giraldi cures it in the same manner, "assigns and stores a device location … within the coordinate frame established by the augmented reality display device 102" ([0016], [0028]) for each device as it is identified ([0031]), likewise predicating that assignment on device-originated output ([0010], [0025]–[0028]). The motivation to combine set forth in the rejection of claims 1 and 3 applies without modification.).
Regarding claim 10, Parashar in view of Giraldi discloses the XR headset of claim 3, wherein the at least one processor is configured to update the device map to include the third connected device mapped based on an image of the third connected device (Parashar identifies a device by imaging it: "the HMD device 104 can recognize a physical pattern (e.g., a QR code or the physical attributes of the device itself, such as a marked model or vendor name, the locations of buttons, and/or device shape) … and compare the recognized pattern against patterns of known physical controlled devices. If a match is made, then the HMD device 104 can download a virtual control object associated with that known physical controlled device" ([0029]; see also [0072], [0088]), the match being "associated with the specific instance of the physical controlled device" ([0029]). The recognition is performed on image data from the headset's outward-facing sensors ([0038]). Parashar does not expressly disclose that the record so updated is a location-indexed device map in a coordinate system of the XR headset — the same deficiency identified in the rejections of claims 1 and 3. Giraldi cures it in the same manner, "assigns and stores a device location...within the coordinate frame established by the augmented reality display device 102" ([0016], [0028]), and likewise predicates that assignment on an image of the device, using "image data analysis to localize the display screen 110 and/or other objects of interest in a coordinate frame of the augmented reality display device 102" ([0013]) and discovering the physical manifestation's location via an image sensor ([0025], [0028]). The motivation to combine set forth in the rejection of claims 1 and 3 applies without modification.).
Regarding claim 14, Parashar in view of Giraldi discloses the XR headset of claim 1, wherein the first gesture of the first type includes a request to trigger an action by the first connected device, and wherein the at least one processor is configured to: send, to the first connected device, the one or more commands to trigger the action by the first connected device (Parashar discloses "Each user action performed through a virtual control causes the HMD device 104 to transmit a remote control instruction through the secure communication connection to the corresponding physical controlled device, which executes the instructed function upon receipt" ([0031]). The gesture is a request for the action performed: the user "can reach forward...and 'touch' the virtual switch control 116 to turn the light 110 on and off," and "'touch' buttons on the virtual panel control 118 to...operate the microwave oven 112" ([0031]); see also [0050] ("the virtual control object executed by the HMD device generates and wirelessly transmits a remote control instruction to the light 306") and [0052] ("the light 306 executes the remote control instruction, such as by turning on or off the light bulb"). Giraldi likewise teaches that a "gesture input location...may be detected as an input for controlling a function of the computing device" ([0030]). The motivation to combine set forth in the rejection of claim 1 applies without modification.).
Claim 15 is rejected under 35 U.S.C. 103 as unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1) for the same reasons set forth with respect to claim 1 above. Claim 15 is directed to a method reciting steps corresponding to the functions recited in claim 1, and the scope and content of the recited limitations are substantially the same, claim 15 differing only in that it omits the memory and processor recitations and recites the XR headset as the frame of reference rather than as the claimed apparatus. Parashar performs the corresponding operations as method steps at the HMD device 104 ([0084]–[0088], methods 900 and 1000), and Giraldi recites its corresponding operations as a method performed on a portable augmented reality computing device ([0023]–[0028], method 400). Accordingly, the teachings of Parashar in view of Giraldi that render claim 1 unpatentable likewise apply to claim 15.
Claim 16 is rejected under 35 U.S.C. 103 as unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1) for the same reasons set forth with respect to claim 2 above. Claim 16 is directed to a method reciting steps corresponding to the functions recited in claim 2, and the scope and content of the recited limitations are substantially the same, claim 16 differing only in statutory class. Accordingly, the teachings of Parashar in view of Giraldi that render claim 2 unpatentable likewise apply to claim 16.
Claim 17 is rejected under 35 U.S.C. 103 as unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1) for the same reasons set forth with respect to claim 3 above. Claim 17 is directed to a method reciting steps corresponding to the functions recited in claim 3, and the scope and content of the recited limitations are substantially the same, claim 17 differing only in statutory class. Accordingly, the teachings of Parashar in view of Giraldi that render claim 3 unpatentable likewise apply to claim 17.
Claim 19 is rejected under 35 U.S.C. 103 as unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1) for the same reasons set forth with respect to claims 7, 8, and 10 above. Claim 19 recites, in the alternative, updating the device map based on user input, information received from the third connected device, or an image of the third connected device — corresponding respectively to the functions recited in claims 7, 8, and 10. Because the alternatives are recited in the alternative, disclosure of any one of them meets the limitation. Claim 19 is directed to a method reciting steps corresponding to the functions recited in those claims, and the scope and content of the recited limitations are substantially the same, claim 19 differing only in statutory class. Accordingly, the teachings of Parashar in view of Giraldi that render claims 7, 8, and 10 unpatentable likewise apply to claim 19.
Claim(s) 9, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi et al. (US 2017/0061692 A1), and further in view of Bleyer et al. (US 2021/0027538 A1).
Regarding claim 9, Parashar in view of Giraldi discloses the XR headset of claim 8, but does not expressly disclose wherein the information received from the third connected device comprises an indication of a change in a location of the third connected device.
Bleyer teaches the device's IMU "can be used to raise a triggering alert to notify the central server when the IOT device 205 has been moved" ([0049]), the device may "send a message or indicator...to inform...that the IOT device may have changed locations" ([0097]). The transmission may consist of nothing more than that indication — "the sensor data may not specifically describe/represent the second location, but instead it may simply provide an indication that the IOT device has been moved" ([0108]; see also [0107]) — and is used to update the stored location, the map being "updated to reflect the new position" automatically ([0096], [0110]). Bleyer further discloses the head-mounted device receiving such data directly from the device ([0072]) "in response to a change in location by the IOT device" (claim 13; see also [0075]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to configure the connected devices of the Parashar/Giraldi system to transmit an indication that their location has changed, and to update the device map on receipt, because Bleyer identifies the problem: portable IOT devices "can be moved from one location to another with relative ease," so "it can be difficult to track and monitor where IOT devices are located," creating "a substantial need to improve how IOT devices are 're-localized'" ([0005]). That problem applies directly to the combined system, in which Giraldi's stored device location is used to interpret a "gesture input location" as a control input (Giraldi [0030]) — an operation that fails if the stored location is stale — and in which Parashar's devices may "enter a low power mode in which they do not continually broadcast beacons" ([0022]), so continuous beaconing cannot be relied on to notice a move. This is the use of a known technique to improve a similar device in the same way, predictably yielding a device map that corrects itself when a device is relocated. All three references are analogous art, directed to maintaining and using locations of networked devices in conjunction with a head-mounted MR/AR device.
Claim 20 is rejected under 35 U.S.C. 103 as unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi (US 2017/0061692 A1), and further in view of Bleyer et al. (US 2021/0027538 A1), for the same reasons set forth with respect to claim 9 above. Claim 20 is directed to a method reciting a step corresponding to the function recited in claim 9, and the scope and content of the recited limitations are substantially the same, claim 20 differing only in statutory class. Accordingly, the teachings of Parashar in view of Giraldi and further in view of Bleyer that render claim 9 unpatentable likewise apply to claim 20.
Claim(s) 11, 12, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Parashar et al. (US 2021/0358294 A1) in view of Giraldi et al. (US 2017/0061692 A1), and further in view of Grohman (US 2010/0106322 A1).
Regarding claim 11, Parashar in view of Giraldi discloses the XR headset of claim 1, wherein the at least one processor is configured to: send, to an additional connected device coupled to the XR headset, a signal configured to trigger an identification output from the additional connected device; determine a location of the additional connected device in the physical environment based on the identification output; map the location of the additional connected device to an additional location in the coordinate system of the XR headset; and update the device map to include the additional connected device at the additional location in the coordinate system of the XR headset (Parashar sends a signal to elicit a localizing output, devices in low power mode "respond to 'wake up' instructions from the HMD device 104 before restarting the beacon broadcasts" ([0022]), the headset receiving the beacon ([0087]) and localizing the device ([0023]). Giraldi's headset is likewise configured to "instruct each device to modulate a physical manifestation...and then detect the modulation...In this manner, the augmented reality display device may determine a device location for each detected device" ([0010]) — instructing the device ([0024], step 408), detecting the manifestation via a location-sensitive input device ([0028], step 420), and "assigns and stores a device location...within the coordinate frame established by the augmented reality display device 102" ([0016], [0028]), the record extended device by device as method 400 "may be repeated for identifying any additional devices" ([0031]).).
However, Parashar in view of Giraldi does not expressly disclose based on a failure to receive the identification output, send, to the additional connected device, one or more additional signals at one or more different times, the one or more additional signals being configured to trigger the identification output from the additional connected device; receive the identification output from the additional connected device in response to at least one of the one or more additional signals.
Grohman addresses messages that solicit a response, defined as "queries" ([0106]), and discloses: "A local controller 290 may be configured to resend a message when a correct reply to the message is not received within the expected timeout period. The timeout period may be set to any non-zero value, e.g., about 1 second. If the message is resent after an initial message, and no response is received within the timeout period after the subsequent message, the local controller 290 may attempt to resend the message again. If a response to the third attempt is not received within the timeout period, the local controller 290 may be configured to cease further resending of the message. Of course, more or fewer attempts may be made" ([0109]). Grohman further discloses repeating a solicitation at intervals until the solicited reply arrives: "the local controller 290 repeats the startup message every 5 minutes until the local controller 290 successfully receives an Equipment Type and Subnet ID assignment" ([0156]); see also [0228] (successive resend delays, "resent up to a predetermined maximum"). Grohman's networked devices are connected home devices of the same general class recited — furnaces, air conditioners, heat pumps, humidifiers, air cleaners, and comfort sensors ([0094], Table I) — communicating over a bus that "may be implemented, e.g., using Bluetooth™, Zibgee or a similar wireless standard" ([0036]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to resend Parashar's wake-up signal or Giraldi's manifestation instruction upon failure to receive the solicited output, as Grohman teaches. Parashar itself establishes that a solicited output may not arrive on the first attempt, since devices "may enter a low power mode in which they do not continually broadcast beacons" ([0022]), and Giraldi's localization cannot proceed unless the instructed manifestation is detected ([0010], [0028]) — so without a retry the device is simply omitted from the map. Grohman supplies the known remedy and its purpose: resending until "a correct reply" is received, and identifying "the unresponsive device" only after the attempts are exhausted ([0109]). This is the use of a known technique to improve a similar device in the same way, predictably yielding localization of a device that fails to respond initially. Grohman is reasonably pertinent to the problem faced by the inventor — obtaining a solicited response from a networked connected device that may not answer on the first request — and is therefore analogous art.
Regarding claim 12, Parashar in view of Giraldi and Grohman discloses the XR headset of claim 11, wherein the at least one processor is configured to: determine the failure to receive the identification output while at least one of the XR headset is within a threshold proximity to the location of the additional connected device or the location of the additional connected device is within a field-of-view of an image sensor of the XR headset (Grohman supplies the failure determination, the controller determining that "a correct reply to the message is not received within the expected timeout period" ([0109]). That determination necessarily occurs while both recited conditions obtain in the combined system. As to proximity: Parashar's localizable beacons "have a limited range that would fill most rooms," rated at "a physical range of 10 m to 100 m" ([0022]), and the headset "may begin to receive such beacons after entering a room containing physical controlled devices" ([0023]) — the wake-up signal being sent, and any reply awaited, while the headset is within that range of the device. Giraldi further tracks the relative position of the headset and the stored device location as "a wearer of the augmented reality display device 102 moves about in the real-world environment 104" ([0017]), so the headset's proximity to a stored device location is known at all times. As to field of view: Giraldi detects the solicited manifestation with "a location-sensitive input device," where "an image sensor may detect light or motion as physical manifestations" ([0028], step 420), the 3D mapping subsystem operating on objects "as these objects are within the user's field of view" (Parashar [0078]) — so the interval during which the manifestation fails to appear is an interval during which the device's location is within the image sensor's field of view. The motivation to combine set forth in the rejection of claim 11 applies without modification.).
Regarding claim 13, Parashar in view of Giraldi and Grohman discloses the XR headset of claim 11, wherein the at least one processor is configured to send the one or more additional signals periodically until at least one of a predetermined amount of time lapses without receiving the identification output from the additional connected device or a number of additional signals is sent without receiving the identification output from the additional connected device (Grohman discloses both alternatives in a single passage. As to sending periodically until a predetermined time lapses: the controller resends "when a correct reply to the message is not received within the expected timeout period," which "may be set to any non-zero value, e.g., about 1 second," and resends again if "no response is received within the timeout period after the subsequent message" ([0109]); see also "the local controller 290 repeats the startup message every 5 minutes until the local controller 290 successfully receives an Equipment Type and Subnet ID assignment" ([0156]). As to a number of signals being sent: "If a response to the third attempt is not received within the timeout period, the local controller 290 may be configured to cease further resending of the message. Of course, more or fewer attempts may be made before ceasing to send the message" ([0109]); see also [0228], the message being "resent up to a predetermined maximum, e.g. 255," after which the controller ceases and "may further be configured to execute the message send/retry cycle again after a predetermined delay period, e.g., about 5 minutes." The motivation to combine set forth in the rejection of claim 11 applies without modification.).
Allowable Subject Matter
Claims 4–6 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable over the prior art of record if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and if the nonstatutory double patenting rejection set forth above is overcome.
The following is a statement of reasons for the indication of allowable subject matter: the prior art of record does not teach or fairly suggest, in combination with the remaining limitations, sending a signal configured to trigger an identification output from one or more connected devices based on a determination that no devices located at a location in the physical environment are mapped to that location in the device map, and updating the device map to include the connected device at that location based on receiving the triggered identification output. Parashar elicits a localizing output in response to the absence of a beacon transmission ([0022]) and conditions localization on whether a particular device has previously registered with the headset ([0023], [0027]); Giraldi initiates its localization routine upon discovering a device's presence over a network (step 404, [0023]). Neither initiates device discovery in response to a determination, keyed to a location in the device map, that a device present at that location is unrecorded.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAJSHEED O BLACK-CHILDRESS whose telephone number is (571)270-7838. The examiner can normally be reached M to F, 10am to 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Quan-Zhen Wang can be reached at (571) 272-3114. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAJSHEED O BLACK-CHILDRESS/Examiner, Art Unit 2685
Prosecution Insights