DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 15-17, 19, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (U.S. Publ. No. 2022/0327775) in view of Kim et al. (U.S. Publ. No. 2019/0164346).
Regarding claim 1, Lee discloses a system and method that uses video to ascertain forces acting on joints of a human. More specifically and as it relates to the claim language Lee discloses a method comprising:
causing a display (see paragraph 0046 where the results are displayed; also see paragraph 0041 and 0053 where a graphic engine renders an image for a video game),
receiving, from the sensor (camera, 212; see Figure 2 and paragraphs 0021 and 0047), motion data of user movements (see paragraphs 0015, 0022, where motion of a user is extracted from the video images), wherein the motion data comprises joint data associated with at least one joint of the user (see paragraphs 0023-0027 where the motion data comprises joint data);
determining, based on the joint data, force information (see paragraphs 0026-0028 where pose data based on the extract joint information is used to determine force information); and
determining, based on the force information, user strength associated with the at least one joint (see paragraph 0040 where the force information is used to determine a user joint load; “one or more internal loads, forces, and/or other biomechanical assessments are generated based on the determined force(s) (e.g., the determined ground reaction force(s) and the determined hand force(s)). In some cases, the act 134 includes an act 136 in which the internal load exerted on one or more body parts, such as a spine, knee, or other joint or portion of the musculoskeletal body, is determined. For example, the hand and ground reaction forces may be input into biomechanical analysis software, such as OpenSim and 3D SSPP, to estimate the internal loads exerted on each body joint”; also see paragraph 0036 where a measure of the body’s capability is ascertained. An internal load for a joint is interpreted as a user strength.)
Lee, however, fails to explicitly disclose the limitation of “based on a calibration of a sensor, to output an avatar of a user within a virtual environment”. While Lee implies the display of an avatar within a virtual environment by implementing the output in a gaming environment, Lee fails to explicitly disclose this. Kim, on the other hand, discloses that it is well known in the art to present an avatar of a user in a virtual environment; and further that it is based on a calibrated sensor. More specifically, Kim discloses providing a realistic augmented reality experience based on video images. Kim discloses capturing video with an RGBD sensor (1000). The video from the sensor (1000) is processed to obtain information about the motions of a user (cybernaut; see Fig. 2) and corresponding skeleton information. This information is used to insert an avatar of the user into the AR content. See Figure 2 and 5. Kim discloses that this feature allows for a realistic augmented reality experience. Furthermore, Kim discloses that the RGBD sensor is calibrated to output the avatar within the virtual environment. See paragraph 0061 – “Herein, in one embodiment, it may be assumed that calibrations such as a relative rotation/movement relation between information representative coordinate systems or sensors according to a camera geometric calibration method of a computer vision field may be performed for an RGB image and a depth image, and skeleton information. Such a geometric calibration assumption may be generally implemented by using various geometric calibration methods of a corresponding field. By using geometric calibration information, a 3D position of a pixel corresponding to a depth image pixel within a color image in association with the depth image pixel may be calculated by using a depth value of each pixel of the depth image.” Kim discloses that this feature allows for accurate reproduction of the user’s avatar with the virtual environment (AR content). See paragraph 0061. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Lee to output an avatar into a virtual environment for a realistic augmented reality experience. Furthermore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Lee calibrate the image sensor before doing so, so that the user’s avatar is accurately reproduced in the virtual environment.
As for claim 2, Kim discloses that the display (display unit, 3000; see Figure 1) is a head mounted display. See paragraph 0028 and 0109. Kim also discloses that the display can be a TV (see paragraph 0002).
With regard to claim 4, while Lee discloses the ability to extract depth data for each pixel (see paragraph 0023), Lee fails to explicity disclose how the z (depth) data is obtained. In other words, Lee fails to disclose that the camera captured depth information using an RGB-D camera. Kim on the other hand, discloses that it is well known in the art to capture depth information using an RGB-D camera. More specifically Kim discloses that the sensor (1000; see Figure 1; and paragraph 0059) is an RGB-D sensor capable of capturing RGB video and depth information. Also see paragraph 0060. Kim implies that this type of sensor allows for the capture of body information including skeleton information. See paragraph 0060. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Lee such that the camera is an RGB-D camera in order to easily extract the skeleton information.
Regarding claim 15, Kim discloses that the calibration of the sensor comprises real-time camera to skeletal pose calibration. See paragraph 0061 where the camera and skeleton information are calibrated to one another. Since the claim does not specify what is meant by “real-time” it is broadly interpreted to mean that the sensor is calibrated to the skeletal pose during operation.
As for claim 16, claim 16 is considered an apparatus claim corresponding substantially to claim 1. Please see the discussion of claim 1 above. Furthermore, Lee discloses an apparatus (system, 200; see Figure 2) comprising; one or more processors (processor, 208, and/or processor 214; see Figure 2), and a memory (memory, 210, and/or memory, 216; see Figure 2) sorting processor-executable instructions, that, when executed by the one or more processors, cause the apparatus to perform the claimed steps. See paragraph 0048 and 0049 and claim 11. Additionally, Kim discloses the use of a processor and a memory for carrying out the steps of the method. See paragraph 0119.
Claim 17 is considered an apparatus claim corresponding substantially to claim 2. Please see the discussion of claim 2 above.
Claim 19 is considered an apparatus claim corresponding substantially to claim 4. Please see the discussion of claim 4 above.
Claim 30 is considered an apparatus claim corresponding substantially to claim 15. Please see the discussion of claim 15 above.
Claim(s) 6, 8, 9, 21, 23, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (U.S. Publ. No. 2022/0327775) in view of Kim et al. (U.S. Publ. No. 2019/0164346) and further in view of Cao et al. (U.S. Publ. No. 2015/0138063).
Regarding claim 6, as mentioned above in the discussion of claim 1, the combination of Lee and Kim disclose all of the limitations of the parent claim. Furthermore, Lee discloses using the joint and motion information in a game setting. See paragraphs 0019, 0041, and 0053. The aforementioned references, however, fail to explicitly disclose that the game allows a user to interact with a virtual object within the virtual environment. Cao, on the other hand, discloses that it is well known in the gaming environment to allow a user to interact with a virtual object within a virtual environment. More specifically Cao discloses a gaming system (home entertainment system, 102; see Figure 1 and paragraph 0014) that allows a user (human subject, 110; see Figure 1 and paragraph 0015) to interact with a virtual object (for example, a virtual balloon, 112; see figure 1 and paragraph 0017) within a virtual environment (see paragraph 0017 where the scene displayed on the display is a virtual environment). One of ordinary skill in the art would recognize that this feature allows for a highly interactive gaming experience and allows for a user to actively engage with objects in the virtual environment. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Lee and Kim to allow for the user to interact with a virtual object in a virtual environment.
As for claim 8, Cao discloses that the magnitude of the motion is determined by tracking joint data, including an angle of at least one joint, while the user interacts with a virtual object; and that the force is determined based on the tracked angle. See Figure 2, where an angle of the joint is tracked to determine how the user is interacting with the virtual object. Also see paragraph 0038 where the position of the virtual skeleton and inherently the angle of the joint is used in determining how the user interacts with the virtual object. Additional as per paragraph 0028 relative position between joints is used to recognize gestures. As indicated by Figure 2, the relative position is indicative of joint angles. Furthermore, Lee discloses monitoring the angles of the joints to determine the internal forces acting on the joint. See paragraphs 0024-0025.
With regard to claim 9, Lee discloses that position data of the joints (including angles of the joints; paragraphs 0024-0025) is used to estimate a force acting on a joint (see paragraph 0035) during a game. As discussed above, Cao discloses interacting with a virtual object in a virtual environment in a game. Thus, when used in combination with Cao, Lee discloses estimating a force acting on the at least one joint while the user interacts with a virtual object.
Claim 21 is considered an apparatus claim corresponding substantially to claim 2. Please see the discussion of claim 2 above.
Claim 23 is considered an apparatus claim corresponding substantially to claim 4. Please see the discussion of claim 4 above.
Claim 24 is considered an apparatus claim corresponding substantially to claim 15. Please see the discussion of claim 15 above.
Claim(s) 10, 12, 13, 25, 27, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (U.S. Publ. No. 2022/0327775) in view of Kim et al. (U.S. Publ. No. 2019/0164346) and further in view of Cao et al. (U.S. Publ. No. 2015/0138063) and Justice et al. (U.S. Publ. No. 2014/0267429).
Regarding claim 10, as mentioned above in the discussion of claim 9. The combination of Lee, Kim, and Cao disclose all of the limitations of the parent claim. Furthermore, as mentioned above in the discussion of claim 9, Lee discloses that position data of the joints (including angles of the joints; paragraphs 0024-0025) is used to estimate a force acting on a joint (see paragraph 0035) during a game. As discussed above, Cao discloses interacting with a virtual object in a virtual environment in a game. Thus, when used in combination with Cao, Lee discloses estimating a force acting on the at least one joint while the user interacts with a virtual object in a gaming environment. The aforementioned references however, fail to explicitly disclose the same steps performed for a second user to determine a second force on a joint of the second user while the second user interacts with a second virtual object; and causing, based on the force information and the second force information, the virtual object to overcome the second virtual object. Justice, on the other hand, discloses that it is well known in the art to allow two gaming competitors to interact with first and second virtual objects to determine a winner of a game. More specifically, Justice discloses a virtual game in which two users may be connected via a networking environment (see Figure 3). Each gaming client (310) captures its own data and sends it to a server which determines how the clients interact in the gaming environment. In an example two users separately interact with two virtual balls to determine a winner of a game. See Figures 4 and 5; and paragraphs 0100-0104. As noted in Figure 2 and paragraph 0059 each client separately processes motion and skeleton data. When used in combination with Lee and Cao, the combination fairly teaches receiving, from a second sensor, second motion data of user movements, wherein the second motion data comprises second joint data associated with at least one joint of a second user; determining, based on the second joint data, second force information, wherein the second force information comprises an estimation of a force acting on the at least one joint of the second user while the second user interacts with a second virtual object within the virtual environment. Regarding the limitation of “causing, based on the force information and the second force information, the virtual object to overcome the second virtual object”, Justice discloses keeping score of the game. Thus, it can be broadly stated that the force information determined, is used to determine if the virtual object associated with the first person “overcomes” the second virtual object of the second person. In other words it can be determined if the first person wins, or scores a point. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Lee to receiving, from a second sensor, second motion data of user movements, wherein the second motion data comprises second joint data associated with at least one joint of a second user; determining, based on the second joint data, second force information, wherein the second force information comprises an estimation of a force acting on the at least one joint of the second user while the second user interacts with a second virtual object within the virtual environment and causing, based on the force information and the second force information, the virtual object to overcome the second virtual object. Justice discloses that such a feature allows for players from different client devices to connect and play together within a multiplayer game. See paragraph 0002.
As for claim 12, as mentioned above in the discussion of claim 9. The combination of Lee, Kim, and Cao disclose all of the limitations of the parent claim. Additionally, at least Cao discloses the ability to output both motion data (optical flow) and data associated with the user (recognized user gestures) interacting with the virtual object for manipulation of an object in a gaming environment. See Figure 2. The aforementioned references, however, fail to explicitly disclose outputting that data to a communication network for remote access. Justice, on the other hand, discloses that it is well known in the art to output motion data and data associated with a user interacting with a virtual object within the virtual environment to a communication network (network, 220; see paragraph 0049) for remote access (via the game service, 230). Justice discloses that such a feature allows for players from different client devices to connect and play together within a multiplayer game. See paragraph 0002. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Lee to allow output the obtained motion data to a communication network for remote access.
With regard to claim 13, Cao discloses that the force data comprises a force perception association with the toutput of the motion data and the data associated with the user interacting with the virtual object within the virtual environment (see paragraphs 0017-0020 where a force magnitude is ascertained from the motion of the user and the skeleton information to apply a force to the virtual object).
Claim 25 is considered an apparatus claim corresponding substantially to claim 10. Please see the discussion of claim 10 above.
Claim 27 is considered an apparatus claim corresponding substantially to claim 12. Please see the discussion of claim 4 above.
Claim 28 is considered an apparatus claim corresponding substantially to claim 13. Please see the discussion of claim 13 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN M VILLECCO whose telephone number is (571)272-7319. The examiner can normally be reached Mon-Thurs 6:00 AM-4:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights