Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 15-26, 28-34 is/are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by Wang U.S. Patent/PG Publication 11361735.
Regarding claim 15 (independent):
A computer-implemented method comprising: (Wang C1 L25-30 A head-mountable device can be worn by a user to display visual information within the field of view of the user. The head-mountable device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system.)
obtaining real-time data derived from monitoring a real-world environment associated with the (VR) headset (Wang C5 L25-30 Alternatively, a system may have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment.)
obtaining virtual display data representing one or more virtual objects within the virtual environment; (Wang C10 L55-67 The display 190 can provide a view of one or more virtual objects 30 that do not correspond to physical objects in the physical environment. For example, the virtual objects 30 can be rendered even when no corresponding physical objects are present in the physical environment. One or more characteristics of the virtual object 30 can be known by the head-mountable device 100 and applied when the virtual object 30 is displayed within the view provided by the display 190. Such virtual object 30 can be displayed as part of a program, at the user's instruction, and/or as automated feedback prompted by a user's activities.)
identifying at least one target object by analyzing at least one of the real-time sensor data or the virtual display data (Wang C11 L15-25 As shown in FIG. 5, the display 190 can provide a view of physical objects 20 and/or virtual objects 30, as well as a portion (e.g., limb) of the user that is within the field of view. One or more physical objects 20 provided in the view of the display 190 can correspond to physical objects in an environment. It will be understood that physical objects 20 can be rendered based on corresponding physical objects in the physical environment, whereas virtual objects 30 can be rendered without corresponding physical objects in the physical environment.)(Wang C10 L1-25 The head-mountable device 100 can perform object recognition with respect to the physical object 20. […] It will be understood that a wide variety of physical objects 20 can be recognized, including items, devices, people, buildings, geographic features, locations, and the like. A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)
classifying an object type of the at least one target object as either a real object or a virtual object; (Wang C11 L20-30 It will be understood that physical objects 20 can be rendered based on corresponding physical objects in the physical environment, whereas virtual objects 30 can be rendered without corresponding physical objects in the physical environment.).
and providing, to the VR headset or another computing device, an indication of the object type during display of the virtual environment. (Wang C11 L25-35 An output can be provided based on one or more conditions. The conditions detected by the head-mountable device 100 can represent when a user would want to react to physical objects 20 and/or virtual objects 30. The output can help a user distinguish between physical objects 20 and virtual objects 30 so that the user can then choose an appropriate reaction (e.g., movement, dodging, interaction, grasping, and/or none of the above).).
Regarding claim 16:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches further comprising:
determining an object position of the at least one target object within the virtual environment (Wang C10 L50-60 For example, recognized objects can be rendered having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the physical environment.) (Wang C11 L15-25 It will be understood that a wide variety of physical objects 20 can be recognized, including items, devices, people, buildings, geographic features, locations, and the like. A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)(Wang C5 L65-C6L5 As another example, a virtual object may adopt a shape or color of a physical article imaged by one or more imaging sensors. As a further example, a virtual object may adopt shadows consistent with the position of the sun in the physical environment.).
wherein indicating the object type is based on the object position. (Wang C10 L45-65 Referring now to FIG. 6, a visual output can be provided as an additive feature included with the virtual object. For example, a visual output 40 can include highlighting, glow, shadow, reflection, outline, border, text, icons, symbols, emphasis, duplication, aura, and/or animation provided with the view of the virtual object 30. Such a visual output 40 can be provided optionally without altering the appearance of the virtual object 30. For example, the visual output 40 can be provided about an outer periphery of the virtual object 30. Additionally or alternatively, the visual output 40 can be provided with partial or entire overlap (e.g., overlaid) with respect to the virtual object 30. The visual output 40 can appear to violate physical laws, such as by casting a shadow that does not appear where it would if the virtual object 30 were a physical object in the physical environment.)(Wang C13 L20-30 Referring now to FIG. 9, a visual output can alter a characteristic of a virtual object. As shown in FIG. 9, a virtual object 30 can be provided with a visual output 46 that alters the size of the virtual object 30. Additionally or alternatively, the visual output 46 can alter other characteristics of the virtual object 30, such as shape, aspect ratio, position, orientation, and the like. The visual output 46 can be applied to an entirety or only a portion of the virtual object 30.).
Regarding claim 17:
The method according to claim 16, has all of its limitations taught by Wang. Wang further teaches further comprising:
determining a distance between the user and the object position within the virtual environment; (Wang C16 L30-50 Clause A: a head-mountable device comprising: a camera configured to capture a view of a physical object in a physical environment; a display configured to display the view of the physical object and a virtual object; and a processor configured to: detect a distance between a user and the virtual object; and when the distance between the user and the virtual object is within a threshold, produce an output indicating to the user that the virtual object is not in the physical environment. Clause B: a head-mountable device comprising: a camera configured to capture a view of a physical object in a physical environment; a sensor configured to detect a distance between a user and the physical object; a display configured to display the view of the physical object and a virtual object; and a processor configured to produce an output indicating to the user that the physical object is in the physical environment when the distance between the user and the physical object is within a threshold.).
and providing the indication based on the determined distance. (Wang C13 L60- C14 L10 The visual output 50 can be provided whenever a particular object and/or type of object is within a view provided by the display 190. Additionally or alternatively, the visual output 50 can be provided whenever a particular object and/or type of object is within a threshold distance of the user and/or has other motion characteristics with respect to the user. Different visual outputs 50 can be provided under different conditions to indicate the respective conditions to the user. It will be understood that general outputs such as visual outputs 50 can be provided with or without other visual outputs described herein.).
Regarding claim 18:
The method according to claim 17, has all of its limitations taught by Wang. Wang further teaches wherein providing the indication comprises indicating the at least one target object when the distance is within a predetermined distance. (Wang C13 L60- C14 L10 The visual output 50 can be provided whenever a particular object and/or type of object is within a view provided by the display 190. Additionally or alternatively, the visual output 50 can be provided whenever a particular object and/or type of object is within a threshold distance of the user and/or has other motion characteristics with respect to the user. Different visual outputs 50 can be provided under different conditions to indicate the respective conditions to the user. It will be understood that general outputs such as visual outputs 50 can be provided with or without other visual outputs described herein.).
Regarding claim 19:
The method according to claim 16, has all of its limitations taught by Wang. Wang further teaches further comprising:
obtaining a tracked object position by tracking the object position of the at least one target object (Wang C5 L5-15 Also, some electronic systems for presenting an MR environment may track location and/or orientation with respect to the physical environment to enable virtual objects to interact with real objects (that is, physical articles from the physical environment or representations thereof). For example, a system may account for movements so that a virtual tree appears stationery with respect to the physical ground.))(Wang C10 L15-24 It will be understood that a wide variety of physical objects 20 can be recognized, including items, devices, people, buildings, geographic features, locations, and the like. A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)(Wang C15 L15-25 For example, where a physical object 20 is identified, whether or not within the field-of-view 90 of the head-mountable device 100, the head-mountable device 100 can determine whether a user is at risk of colliding with the physical object 20. Such a determination can be based on a trajectory of the physical object 20 and/or the user. If one or both trajectories end in a collision, an output can be provided, as described herein.).
and updating the indication based on the tracked object position (Wang C13 L45-50 As shown in FIG. 10, a virtual object 30 can be displayed a moving toward a user. While an initial trajectory of the virtual object 30 may indicate that it would collide with the user, the trajectory can be altered to allow the user to recognize the object as being virtual. By further example, as shown in FIG. 11, a trajectory of the virtual object 30 can be altered in apparent violation of physical laws (e.g., contrary to the effects of gravity). Such motion contrary to physical laws in the physical environment can allow the user to recognize the object as being virtual, and the user can react accordingly.)(Wang C14 L25-40 Sound can be emitted in a manner that is based on the corresponding detections. For example, the volume, pitch, duration, and/or other characteristics of the sound can be based on detected characteristics (e.g., proximity, etc.) and updated accordingly. Where separate sounds are emitted, the number of sounds, time between sounds, duration of each sound, and/or variations thereof can be provided based on detected characteristics (e.g., proximity, etc.) and updated accordingly.).
Regarding claim 20:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches providing the indication when the at least one target object is classified as a virtual object. (Wang C16 L50-60 a first mode in which a visual output is applied to the virtual object and not applied to the physical objects; or a second mode in which the visual output is not applied to the virtual object.)(Wang C16 L65-68 Clause 2: the processor is further configured to apply the visual output to the virtual object in the view without applying the output to the physical object in the view.).
Regarding claim 21:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches providing the indication when the at least one target object is classified as a real object. (Wang C16 L50-60 a first mode in which a visual output is applied to the virtual object and not applied to the physical objects; or a second mode in which the visual output is not applied to the virtual object.)(Wang C16 L65-68 Clause 2: the processor is further configured to apply the visual output to the virtual object in the view without applying the output to the physical object in the view.).
Regarding claim 22:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches wherein the indication is generated based on a preset toggled state selected by the user prior to the display of the virtual environment. (Wang C11 L25-35 The output can help a user distinguish between physical objects 20 and virtual objects 30 so that the user can then choose an appropriate reaction (e.g., movement, dodging, interaction, grasping, and/or none of the above).)(Wang C15 L40-60 In some embodiments, a head-mountable device can provide outputs according to a mode of operation. For example, in a first mode, an output can be applied to virtual and/or physical objects according to the description provided herein. In a second mode, such outputs can be omitted regardless of detected conditions relating to the objects. In the second mode, the user can be provided with a more immersive experience that is not interrupted by outputs that help distinguish between virtual and physical objects. The selection between the first mode and the second mode can be managed by user input, so that the desired experience is provided according to user desires. By further example, the selection between the first mode and the second mode can be automated and based on one or more of a variety of conditions, such as time, location, detected objects, user identity, prior usage, and the like.)(Wang C2 L50-60 While a head-mountable device can provide outputs to a user in a variety of ways, it can also be helpful to allow a head-mountable device to receive inputs from a user.).
Regarding claim 23:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches wherein providing the indication comprises at least one of:
displaying a marker at the object position associated with the at least one target object (Wang C10 L45-65 Referring now to FIG. 6, a visual output can be provided as an additive feature included with the virtual object. For example, a visual output 40 can include highlighting, glow, shadow, reflection, outline, border, text, icons, symbols, emphasis, duplication, aura, and/or animation provided with the view of the virtual object 30. Such a visual output 40 can be provided optionally without altering the appearance of the virtual object 30.)
outputting an auditory signal (Wang C14 L5-30 Referring now to FIG. 13, a head-mountable device can provide sound or other audio or acoustic output to a user to enhance a user's awareness of physical and/or virtual objects. For example, the head-mountable device 100 can operate a speaker thereof or a speaker of another device to output sound to the user. The sound can be emitted based on one or more of a variety of detected conditions.).
or outputting a haptic signal (Wang C14 L45-55 Referring now to FIG. 14, a head-mountable device can provide haptic feedback to a user to enhance a user's awareness of physical and/or virtual objects. For example, the head-mountable device 100 can operate a haptic feedback device and/or communication with one or more external devices 200 that include haptic feedback components. The haptic feedback can be provided based on one or more of a variety of detected conditions. For example, the haptic feedback can be provided based on detection of a physical object 20 and/or one or more other characteristics of the physical object 20, such as distance from the user, velocity, acceleration, and/or other information.).
Regarding claim 24:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches wherein identifying at least one target object further comprises deriving object information associated with the at least one target object. (Wang C10 L1-25 The head-mountable device 100 can perform object recognition with respect to the physical object 20. […] It will be understood that a wide variety of physical objects 20 can be recognized, including items, devices, people, buildings, geographic features, locations, and the like. A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)
Regarding claim 25:
The method according to claim 24, has all of its limitations taught by Wang. Wang further teaches wherein the object information comprises one or more of
an object label (Wang C10 L1-25 A characteristic of the physical object 20 can include an identity, name, type, reference)
an object orientation (Wang C10 L50-55 For example, recognized objects can be rendered having features (e.g., position, orientation, color, size, etc.) that are based on detections of the physical objects in the physical environment.).
an object position; an object dimension; or an object material (Wang C10 L1-25 A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)
Regarding claim 26:
The method according to claim 24, has all of its limitations taught by Wang. Wang further teaches wherein the indication is further based on the object information to provide additional attributes regarding the at least one target object (Wang C10 L1-25 The head-mountable device 100 can perform object recognition with respect to the physical object 20. For example, the view can be captured and analyzed by the head-mountable device 100 to determine a characteristic of the physical object 20. Object recognition can be facilitated by a camera of the head-mountable device 100. Additionally or alternatively, object recognition can be facilitated by one or more other sensors of the head-mountable device 100. For example, the head-mountable device 100 can be in communication with another device that performs object recognition and communicates results to the head-mountable device 100. By further example, the head-mountable device 100 can be in communication with a physical object 20 that includes communication circuitry to communicate information to the head-mountable device 100. […] It will be understood that a wide variety of physical objects 20 can be recognized, including items, devices, people, buildings, geographic features, locations, and the like. A characteristic of the physical object 20 can include an identity, name, type, reference, color, size, shape, distance, position, velocity, acceleration, make, model, or other feature detectable by the head-mountable device 100.)
Regarding claim 13:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches wherein obtaining real-time sensor data comprises obtaining at least one of video data (Wang C8 L1-10 It will also be understood that the head-mountable device 100 can include any number of cameras. The cameras can be positioned and oriented to capture different views. For example, one camera can capture an image of an object from one perspective and another camera can capture an image of an object from another perspective. Additionally or alternatively, the other camera can capture an image of an object that is not captured by the first camera.).
or sensor data of the real-world environment. (Wang 8 L35-50 The head-mountable device 100 can include one or more other sensors. Such sensors can be configured to sense substantially any type of characteristic such as, but not limited to, images, pressure, light, touch, force, temperature, position, motion, and so on. For example, the sensor can be a photodetector, a temperature sensor, a light or optical sensor, an atmospheric pressure sensor, a humidity sensor, a magnet, a gyroscope, an accelerometer, a chemical sensor, an ozone sensor, a particulate count sensor, and so on. By further example, the sensor can be a bio-sensor for tracking biometric characteristics, such as health and activity metrics.)
Regarding claim 29 (independent):
The claim is a parallel version of claim 15. As such it is rejected under the same teachings.
Regarding claim 30:
The claim is a parallel version of claim 16. As such it is rejected under the same teachings.
Regarding claim 31:
The claim is a parallel version of claim 17. As such it is rejected under the same teachings.
Regarding claim 32:
The claim is a parallel version of claim 18. As such it is rejected under the same teachings.
Regarding claim 33:
The claim is a parallel version of claim 19. As such it is rejected under the same teachings.
Regarding claim 34:
The claim is a parallel version of claim 15. As such it is rejected under the same teachings.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang U.S. Patent/PG Publication 11361735 in view of Soryal U.S. Patent/PG Publication 20230022194.
Regarding claim 27:
The method according to claim 15, has all of its limitations taught by Wang. Wang further teaches wherein at least one of identifying the target object or classifying the object type comprises utilizing (Wang C10 L1-25 The head-mountable device 100 can perform object recognition with respect to the physical object 20. For example, the view can be captured and analyzed by the head-mountable device 100 to determine a characteristic of the physical object 20. Object recognition can be facilitated by a camera of the head-mountable device 100. Additionally or alternatively, object recognition can be facilitated by one or more other sensors of the head-mountable device 100. For example, the head-mountable device 100 can be in communication with another device that performs object recognition and communicates results to the head-mountable device 100. By further example, the head-mountable device 100 can be in communication with a physical object 20 that includes communication circuitry to communicate information to the head-mountable device 100.) (Wang C11 L20-30 It will be understood that physical objects 20 can be rendered based on corresponding physical objects in the physical environment, whereas virtual objects 30 can be rendered without corresponding physical objects in the physical environment.).
Wang does not teach machine learning. In a related field of endeavor, Soryal teaches:
wherein at least one of identifying the target object or classifying the object type (Soryal [0058] In embodiments, the processing system 240 includes an artificial intelligence or machine learning module. The processing system 240, in conjunction with the artificial intelligence or machine learning module, operates to identify an unrecognized situation for the user 202. An unrecognized situation is one in which the user 202 may be immersed in an experience and unable to discern real objects, characters and events from virtual objects, characters and events.)(Soryal [0070] In some embodiments, the analytics may use artificial intelligence or machine learning (collectively referred to as AI/ML) to identify virtual or real objects or persons in the immersive experience.)
comprises utilizing a machine learning algorithm (Soryal [0035] The system 200 may draw on learned information available to the system 200, such machine learning solutions, to understand the status of the user 202 as well as to understand the environment of the user 202 and objects in the environment.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use machine learning as taught by Soryal. The rationale for doing so would have been that it is a simple substitution of one known element for another to obtain predictable results since Wang uses a computer for object recognition with an undisclosed algorithm and Soryal uses a computer for object recognition with machine learning, where there are predictable result since it is the same input and the same output, merely with a different algorithm. Therefore it would have been obvious to combine Soryal with Wang to obtain the invention.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights