CTNF 18/490,003 CTNF 84203 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-4, 6-9, and 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaifosh et al. (US 20230072423 A1) . Regarding claim 1, 13, and 15 , Kaifosh et al. disclose an apparatus, comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform (implemented in software, code comprising the software can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers, [0344], [0345]) a method, comprising ([0346]) and non-transitory program storage device readable with an apparatus, tangibly embodying a program of instructions that when executed with the apparatus, cause the apparatus to perform at least ([0345]): obtaining LiDAR data from a mediated reality device wherein the mediated reality device is used with a user and the LiDAR data comprises data points representing at least part of an object (For example, augmented-reality system 1100 and/or virtual-reality system 1200 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor, [0368], XR system 391001 or the neuromuscular activity system 391002 may include one or more auxiliary sensor(s), Kinect, [1732], In various embodiments of this implementation, the information about the objects in the environment, captured by the at least one camera, may comprise images, with each of the images being formed of an array of pixels. Each pixel of the array of pixels may comprise depth data and visual data, [1755]); obtaining position data relating to a position of the user's hand relative to the at least part of the object (measure forearm orientation and the neuromuscular signals being used to determine hand and wrist configuration and forces, tracking one or more position(s) of articulated rigid bodies, [0615], distinguish between whether someone is moving freely or pressing against object(s) and/or surface(s) (which may include another part of the user's body), determine which object(s) and/or surfaces is or are being interacted with, which position(s) on the surface(s) and/or object(s) are being touched, [0616]); obtaining interaction data indicating how the user's hand is interacting with the at least part of the object wherein at least some of the interaction data is obtained from an electromyography device coupled to the user, [0615], For instance, camera data may be used to distinguish between whether someone is moving freely or pressing against object(s) and/or surface(s) (which may include another part of the user's body), determine which object(s) and/or surfaces is or are being interacted with, which position(s) on the surface(s) and/or object(s) are being touched, and can assist with estimating a skeletal configuration, position, and/or force, [0616], When an XR user is applying self force(s) against the user's own body (e.g., the user is pinching his or her own arm), the camera can assist with determining the skeletal configuration and/or position (e.g. which joint segments are involved in touching the arm), and the neuromuscular sensors can be used to determine the intensity of the force, [0617]); and using the position data to map the interaction data to the LiDAR data to enable the interaction data to be used to classify the at least part of the object (“In some embodiments, data from one or more cameras may be used to determine the position of an arm, a hand, a forearm, or another part of the user's body. Also, camera data may be used to combat drift in an IMU-based estimate of forearm position, with the IMU information being used to measure forearm orientation and the neuromuscular signals being used to determine hand and wrist configuration and forces. In this embodiment, positional tracking reference marks on a band of neuromuscular sensors (e.g., a band of EMG sensors) may be used, especially when the camera is used to refine the IMU-based system for tracking one or more position(s) of articulated rigid bodies. As will be appreciated, data from a single camera may be used or data from two or more cameras may be used. According to some embodiments, camera data may be used for determining whether an object (e.g., a hand, finger, or another physical object) is subjected to a force. For instance, camera data may be used to distinguish between whether someone is moving freely or pressing against object(s) and/or surface(s) (which may include another part of the user's body), determine which object(s) and/or surfaces is or are being interacted with, which position(s) on the surface(s) and/or object(s) are being touched, and can assist with estimating a skeletal configuration, position, and/or force. It is appreciated that although camera data can be used to determine whether a force is being applied, camera data is not particularly suited to determining a magnitude of the force(s) applied. To this end, other input signals (e.g., neuromuscular signals) can be used to determine an amount of force applied and also assist with determining the skeletal configuration and/or position. When an XR user is applying self force(s) against the user's own body (e.g., the user is pinching his or her own arm), the camera can assist with determining the skeletal configuration and/or position (e.g. which joint segments are involved in touching the arm), and the neuromuscular sensors can be used to determine the intensity of the force. In this way, a more accurate representation of the arm position(s) and/or the hand position(s) and force(s) can be constructed” [0615]-[0617]). The embodiment described in the last limitation combines camera data with neuromuscular sensor data. As the previous paragraphs describe that LIDAR is one form of sensor or camera data that is obtained, it would have been obvious at the time of filing to one of ordinary skill in the art the embodiment described in paragraphs 615-617 could be using a LIDAR camera as opposed to another type of camera. Regarding claims 2 and 14 , Kaifosh et al. disclose an apparatus and method as claimed in claims 1 and 13. Kaifosh et al. further indicate the classification of the at least part of the object comprises at least one of: a type of the at least part of the object, a function of the at least part of the object, or a configuration of the at least part of the object (determine which object(s) and/or surfaces is or are being interacted with, which position(s) on the surface(s) and/or object(s) are being touched, [0616]) Regarding claim 3 , Kaifosh et al. disclose an apparatus as claimed in claim 1. Kaifosh et al. further indicate the classification of the at least part of the object is used to control rendering of one or more mediated reality functions with the mediated reality device (trained inference model may output data useable for applications such as applications for rendering a representation of the user's body in an XR environment, [0266], visual rendering of the force may be displayed within the AR environment to let the user or another user in a shared AR environment visualize an amount of force being applied to the object, [0330], musculoskeletal representation is updated in real time and a visual representation of a hand (e.g., within an extended reality environment) may be rendered based on the current handstate estimates, [0618]). Regarding claim 4 , Kaifosh et al. disclose an apparatus as claimed in claim 3. Kaifosh et al. further indicate the instructions, when executed with the at least one processor, cause the apparatus to perform at least one of: controlling a respective position of one or more displayed mediated reality images relative to the position of the at least part of the object, controlling scaling of one or more mediated reality images relative to the position of the at least part of the object, displaying one or more mediated reality images relating to the user's interaction with the at least part of the object, providing a user input region at a location relative to the at least part of the object or preventing user inputs at a location relative to at the at least part of the object (The method may comprise: receiving, as input, the plurality of neuromuscular signals sensed from the user by the plurality of neuromuscular sensors; determining, based at least in part on the plurality of neuromuscular signals, information relating to an interaction of the user with the physical object in the AR environment generated by the AR system; and instructing the AR system to provide feedback based, at least in part, on the information relating to an interaction of the user with the physical object, [0346], As described herein, the user's interaction with one or more physical objects in the AR environment can take many forms, including but not limited to: selection of one or more objects, control of one or more objects, activation or deactivation of one or more objects, adjustment of settings or features relating to one or more objects, etc. As will be appreciated, the user's interaction may take other forms enabled by the AR system for the environment, and need not be the interactions specifically listed herein, [0621], XR-based system, the 3D map corresponding to the reference object may be accessed, and control interfaces for smart devices in an environment corresponding to the 3D map may be activated, [1740]). Regarding claim 6 , Kaifosh et al. disclose an apparatus as claimed in claim 4. Kaifosh et al. further indicate the mediated reality images comprise information relating to the classification of the at least part of the object (Some conventional AR systems include camera-based technologies that are used to identify and map physical objects in the user's real-world environment, [0259], At act 510, physical objects in the AR environment are identified using one or more cameras (e.g., the camera(s) 204) associated with the AR-based system (e.g., the system 200). Images captured by camera(s) may be particularly useful in mapping environments and identifying physical objects in an AR environment. For example, if the user of the AR-based system is located in the user's kitchen, the camera(s) associated with the AR-based system may detect multiple physical objects in the user's AR environment, such as a refrigerator, a microwave oven, a stove, a pen, or an electronic device on a kitchen counter, [0339], The process 500 then proceeds to act 512, where an interaction of the user with a physical object in the AR environment is identified. The identification of the interaction may be made, at least in part, based on one or more images captured by the camera(s) associated with the AR system. For example, it may be determined from the one or more images that the user is holding a physical object (e.g., a writing implement), touching an object (e.g., a surface of a table) or reaching toward an object (e.g., a thermostat on the wall)., [0340], “The present technology disclosed herein provides mapping systems and mapping methods that enable a user to create an electronic 3D map of an environment through a combination of neuromuscular sensing technology and imaging technology. A 3D map may be generated in which objects in the environment are mapped. As described below, the 3D map may include image information as well as location and depth information for the objects. The 3D map also may include additional information, e.g., information identifying which of the objects is a remotely controllable object, i.e., a smart device. The 3D map also may include self-identification information, in which an object in the environment may serve as a reference object for the environment and also may serve as a searchable object used to identify a 3D map corresponding to the environment”, [1670], a third activation state may be recognized by the processor(s) 39112 as the user's desire to identify a specific object (e.g., a smart device or controllable object amongst other objects), [1724], Of the various real-world objects in the captured video/images, the user may identify one or more controllable object(s) (i.e., smart device(s)), which may be controlled remotely as so-called “Internet of Things” (IoT) object(s). For example, the room may include an IoT controllable lamp, an IoT controllable sound system, and an IoT controllable videogame monitor, [1739]) [classification interpreted as identifying type of object and whether or not it is controllable]. Regarding claim 7 , Kaifosh et al. disclose an apparatus as claimed in claim 1. Kaifosh et al. further indicate the position data is obtained from at least one of an inertial measurement unit or LiDAR data (The one or more sensor(s) 110 may include one or more auxiliary sensor(s), such as one or more Inertial Measurement Unit(s) or IMU(s), [0278], Sensor 1140 may represent one or more of a variety of different sensing mechanisms, such as a position sensor, an inertial measurement unit (IMU), a depth camera assembly, a structured light emitter and/or detector, or any combination thereof, [0352] For example, augmented-reality system 1100 and/or virtual-reality system 1200 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor. An artificial-reality system may process data from one or more of these sensors to identify a location of a user, to map the real world, to provide a user with context about real-world surroundings, and/or to perform a variety of other functions, [0368], In some embodiments, data from one or more cameras may be used to determine the position of an arm, a hand, a forearm, or another part of the user's body. Also, camera data may be used to combat drift in an IMU-based estimate of forearm position, with the IMU information being used to measure forearm orientation and the neuromuscular signals being used to determine hand and wrist configuration and forces. In this embodiment, positional tracking reference marks on a band of neuromuscular sensors (e.g., a band of EMG sensors) may be used, especially when the camera is used to refine the IMU-based system for tracking one or more position(s) of articulated rigid bodies, [0615]). Regarding claim 8 , Kaifosh et al. disclose an apparatus as claimed in claim 1. Kaifosh et al. further indicate the interaction data is also obtained from at least one or more of: wearable strain gauges coupled to a part of the user or light sensors (neuromuscular sensors may be arranged circumferentially on an adjustable and/or elastic band, such as a wristband or an armband structured to be worn around a user's wrist or arm, [0282], The statistical model may be used to predict the handstate information based on IMU signals, neuromuscular signals (e.g., EMG, MMG, and SMG signals), external device signals (e.g., camera or laser-scanning signals), or a combination of IMU signals, neuromuscular signals, and external device signals detected as a user performs one or more movements, [0447], positional tracking reference marks on a band of neuromuscular sensors (e.g., a band of EMG sensors) may be used, especially when the camera is used to refine the IMU-based system for tracking one or more position(s) of articulated rigid bodies, [0615], the EMG sensors may be the sensors 704 arranged on the band 702, as shown in FIG. 7A; in some embodiments, the EMG sensors may be the sensors 810 arranged on the elastic band 820, as shown in FIG. 8A. The gestures performed by the user may include static gestures, such as placing the user's hand palm down on a table; dynamic gestures, such as waving a finger back and forth; and covert gestures that are imperceptible to another person, such as slightly tensing a joint by co-contracting opposing muscles, pressing on an object or surface, [0622]). Regarding claim 9 , Kaifosh et al. disclose an apparatus as claimed in claim 1. Kaifosh et al. further indicate the interaction with the at least part of the object comprises at least one of: the user touching the at least part of the object, the user applying a force to the at least part of the object, the user moving the at least part of the object, the user holding the at least part of the object, or the user gripping the at least part of the object (determine a force (e.g., a grasping force) applied to a physical object, [0261], placing or pressing the palm of a hand down on a solid surface or grasping a ball, continuous gestures, such as waving a finger back and forth, grasping and throwing a ball, or a combination of discrete and continuous gestures, [0612], which position(s) on the surface(s) and/or object(s) are being touched, and can assist with estimating a skeletal configuration, position, and/or force, [0616], allowing for the capture and detection of subtle, small, or fast movements and/or variations in force exerted by a user (e.g., varying amounts of force exerted through a stylus, writing instrument, or finger being pressed against a surface), [0644], tapping or typing actions performed by the user (e.g., tapping on a surface of a physical keyboard, tapping on a surface that has a virtual keyboard projected thereon by the XR system, tapping on a surface that does not have a keyboard projected on it, [0646]). Regarding claim 11 , Kaifosh et al. disclose an apparatus as claimed in claim 1. Kaifosh et al. further indicate a device comprising an apparatus as claimed in claim 1 (AR glasses or another viewing device) that provides AR information, [0306], Generally, an XR system such as the AR system 201 may take the form of a pair of goggles or glasses or eyewear, or other type of display device that shows display elements to a user that may be superimposed on the user's “reality.”, [0314], Oculus Quest, Oculus Rift S, and Spark AR Studio available from Facebook (Menlo Park, Calif., USA); or any other type of AR or other XR device, [0315] To this end, the neuromuscular activity system 202 may be capable of providing output to other parts of the AR-based system 200 for the purpose of displaying or generating feedback output. In some implementations, the AR-based system 200 may have one or more audio outputs, displays, indicators and/or other type of output device capable of providing or rendering feedback to the user, [0632]). Regarding claim 12 , Kaifosh et al. disclose an apparatus as claimed in claim 11. Kaifosh et al. further indicate the device comprises at least one of: a mediated reality device or a user electronic device (AR glasses or another viewing device) that provides AR information, [0306], Generally, an XR system such as the AR system 201 may take the form of a pair of goggles or glasses or eyewear, or other type of display device that shows display elements to a user that may be superimposed on the user's “reality.”, [0314], Oculus Quest, Oculus Rift S, and Spark AR Studio available from Facebook (Menlo Park, Calif., USA); or any other type of AR or other XR device, [0315] To this end, the neuromuscular activity system 202 may be capable of providing output to other parts of the AR-based system 200 for the purpose of displaying or generating feedback output. In some implementations, the AR-based system 200 may have one or more audio outputs, displays, indicators and/or other type of output device capable of providing or rendering feedback to the user, [0632]) . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim s 5 and 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following art is cited as relevant to claim 5: US 20210406697 A1 : game content to be presented may include various types of content, as may include virtual reality (VR), augmented reality (AR), image, textual, audio, haptic, or video content. In at least one embodiment, client device 202 may include or comprise a device such as a desktop computer, notebook computer, gaming console, smart phone, tablet computer, VR headset, AR goggles, a wearable computer, or a smart television. In at least one embodiment, determination of an object can enable interactions to be determined from this schema. In at least one embodiment, data from feature vectors adhering to this schema can be encoded into a latent space. US 20220237838 A1 : this image content can relate to gaming, virtual reality (VR), mixed reality (MR), or augmented reality (AR) applications. In at least one embodiment, client device 502 may include or comprise a device such as a desktop computer, notebook computer, gaming console, smart phone, tablet computer, VR headset, AR goggles, a wearable computer, a digital camera, or a smart television. In at least one embodiment, determination of an object can enable interactions and relationships to be determined from this schema. In at least one embodiment, data from feature vectors adhering to this schema can be encoded into a latent space. US 20250336134 A1 : In one or more embodiments, the trained models (e.g., the trained hand pose estimation model and/or the trained hand gesture recognition model) may be incorporated in an extended reality (XR) device, such as an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality device. 1. A method comprising: generating, by a processor, from a first conditional variational autoencoder comprising a first latent space and a first transformer decoder, a set of finger poses; generating, by the processor, from a second conditional variational autoencoder comprising a second latent space and a second transformer decoder, a set of wrist motions; and combining, by the processor, the set of finger poses and the set of wrist motions to generate a synthetic dataset of hand and arm gestures. The following art is cited as relevant to claim 10: US 20150379238 A1 [closest art]: In an example, an eating-related gesture can be recognized based on the configuration and movement of a person's thumb and index finger, as well as interaction among the thumb, index finger, and a portion of food. In an example, an eating-related gesture can be recognized based on the configuration and movement of a person's thumb and index finger, as well as interaction between the thumb, finger, and a food-transporting object (such as a fork, spoon, chop stick, drinking glass, beverage can, cup, mug, or bowl). In an example, an eating-related gesture can be recognized based on the configuration and movement of a person's thumb and index finger, as well as interaction among the thumb, finger, a portion of food, and a food-transporting object (such as a fork, spoon, chop stick, drinking glass, beverage can, cup, mug, or bowl). In an example, an eating-related gesture can be recognized based on the configuration and movement of a person's thumb and index finger, as well as interaction between the thumb and finger and a food-transporting object, and also interaction between the food-transporting object and a portion of food. US 20190087966 A1 : As described herein, the interaction recognition system 10 may be configured to identify various objects based on a detected interaction between the objects and a part or portion of a person's body US 20190286892 A1 : A method that includes one or more processing devices performing operations comprising: accessing, from a memory device, an input image; transforming the input image by applying human-object interaction metadata to the input image that identifies a part of a human depicted in the input image being in contact with a part of an object depicted in the input image, wherein applying the human-object interaction metadata comprises: providing the input image to an interaction detection network having a pose estimation subnet, an object contact subnet, and an interaction-detection subnet that receives outputs of the pose estimation subnet and the object contact subnet, computing a joint-location heat map by applying the pose estimation subnet to the input image, the joint-location heat map identifying one or more human joint locations in the input image, computing a contact-point heat map by applying the object contact subnet to the to the input image, the contact-point heat map identifying one or more contact points on the object depicted in the input image, and generating the human-object interaction metadata by applying the interaction-detection subnet to the joint-location heat map and the contact-point heat map, wherein the interaction-detection subnet is trained to identify an interaction based on joint-object contact pairs, wherein a joint-object contact pair comprises a relationship between a human joint location and a contact point; and providing an image search system with access to the input image having the human-object interaction metadata. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M ENTEZARI HAUSMANN whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671 Application/Control Number: 18/490,003 Page 2 Art Unit: 2671 Application/Control Number: 18/490,003 Page 3 Art Unit: 2671 Application/Control Number: 18/490,003 Page 4 Art Unit: 2671 Application/Control Number: 18/490,003 Page 5 Art Unit: 2671 Application/Control Number: 18/490,003 Page 6 Art Unit: 2671 Application/Control Number: 18/490,003 Page 7 Art Unit: 2671 Application/Control Number: 18/490,003 Page 8 Art Unit: 2671 Application/Control Number: 18/490,003 Page 9 Art Unit: 2671 Application/Control Number: 18/490,003 Page 10 Art Unit: 2671 Application/Control Number: 18/490,003 Page 11 Art Unit: 2671 Application/Control Number: 18/490,003 Page 12 Art Unit: 2671 Application/Control Number: 18/490,003 Page 13 Art Unit: 2671 Application/Control Number: 18/490,003 Page 14 Art Unit: 2671 Application/Control Number: 18/490,003 Page 15 Art Unit: 2671
Prosecution Insights