DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-6 and 8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5-6 recite “wherein the generating of the mixed image”. Here the phrase “the mixed image” has a lack of antecedent basis.
Claim 5 recites “wherein the generating of the mixed image includes making, in a case where the reference position is set to the body part of the first users, position information of the body part of the first user and position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of the real video and the virtual object”. Here the limitation is not interpretable because it is not clear what is being made.
Claim 6 recites “wherein the generating of the mixed image includes making, in a case where the reference position is set to the attention object, position information of the attention object and the position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of the real video and the virtual object”. Here the limitation is not interpretable because it is not clear what is being made.
Claim 8 recites “the line of sight”. Here the phrase has a lack of antecedent basis.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-5, 7-8, 11-13 and 17-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Assouline et al. (US Pat. Pub. No. 20220157000 “Assouline”).
Regarding claim 1 Assouline teaches A video processing apparatus that is used by a first user and processes a video, comprising one or more processors and/or circuitry (“[0018] FIG. 1 is a block diagram showing an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of a client device 102, each of which hosts a number of applications, including a messaging client 104 and other external applications 109 (e.g., third-party applications). [0178] The machine 1300 may include processors 1304, memory memory/storage 1306, and I/O components 1318, which may be configured to communicate with each other such as via a bus 1302”) configured to:
acquire a real video of a real space visually recognized by the first user (“[0039] The augmentation system 208 provides functionality to generate, display, and track virtual objects at positions relative to a real-world object (e.g., a person) depicted in a video captured by the client device 102”);
transmit the acquired real video to another video processing apparatus used by a second user different from the first user (“[0019]…..The data exchanged between messaging clients 104, and between a messaging client 104 and the messaging server system 108, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video or other multimedia data)”);
receive motion information concerning motion of the second user from the other video processing apparatus (“[0129] At operation 701, the augmentation system 208 implemented by a client device 102 associated with a first user receives a communication from a second user. For example, a client device 102 of a second user transmits a chat message to a first user that attaches or includes a movement vector recorded by the second user.”);
generate a virtual object of a body part of the second user, which can be displayed in the video, based on the received motion information (“[0132] At operation 704, the augmentation system 208 animates the 3D avatar to mimic the 3D movement of the set of skeletal joints of the second user. [0133]…..In one example, the augmentation system 208 maintains a set of templates to generate virtual objects to be displayed in the video. Upon receiving a selection of a template from among the set of templates, and a selection of a position in the video, the augmentation system 208 generates and assigns the virtual object to the position within the three-dimensional space of the video”); and
generate a mixed video by mixing the real video and the virtual object, and wherein the generating of the mixed video includes generating of the mixed video by aligning respective positions of the real video and the virtual object when generating the mixed video (“[0153] At operation 804, the augmentation system 208 detects a 3D reference point of a real-world object depicted in a video (e.g., a real-time video feed being captured by a client device 102). For example, the augmentation system 208 selects one or more skeletal joints of a person depicted in the video and computes a 3D coordinate of the selected skeletal joints as the 3D reference point.
[0154] At operation 806, the augmentation system 208 orients the virtual object based on the 3D reference point”).
Claim 17 is directed to a method claim and its steps are similar in scope and functions performed by the apparatus claim1 and therefore claim 17 is also rejected with the same rationale as specified in the rejection of claim 1.
Claim 18 is directed to a non-transitory computer-readable storage medium claims (“[0013] FIG. 13 is a block diagram illustrating components of a machine able to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein, according to example embodiments”) and its elements are similar in scope and functions performed by the apparatus claim1 and therefore claim 18 is also rejected with the same rationale as specified in the rejection of claim 1.
Regarding claim 2 Assouline teaches wherein the generating of the mixed video includes generating of the mixed video by setting a predetermined position as a common reference position and aligning the respective positions of the real video and the virtual object when generating the mixed video (“[0153] At operation 804, the augmentation system 208 detects a 3D reference point of a real-world object depicted in a video (e.g., a real-time video feed being captured by a client device 102). For example, the augmentation system 208 selects one or more skeletal joints of a person depicted in the video and computes a 3D coordinate of the selected skeletal joints as the 3D reference point.
[0154] At operation 806, the augmentation system 208 orients the virtual object based on the 3D reference point”).
Regarding claim 3 Assouline teaches wherein the one or more processors and/or circuitry is/are further configured to set the reference position. (“[0153] At operation 804, the augmentation system 208 detects a 3D reference point of a real-world object depicted in a video (e.g., a real-time video feed being captured by a client device 102). For example, the augmentation system 208 selects one or more skeletal joints of a person depicted in the video and computes a 3D coordinate of the selected skeletal joints as the 3D reference point.
[0154] At operation 806, the augmentation system 208 orients the virtual object based on the 3D reference point”).
Regarding claim 4 Assouline teaches wherein the setting includes setting the reference position to a body part of the first user or an attention object which is positioned on a line of sight of the second user and gazed by the second user. (“[0153] At operation 804, the augmentation system 208 detects a 3D reference point of a real-world object depicted in a video (e.g., a real-time video feed being captured by a client device 102). For example, the augmentation system 208 selects one or more skeletal joints of a person depicted in the video and computes a 3D coordinate of the selected skeletal joints as the 3D reference point”).
Regarding claim 5 Assouline teaches wherein the generating of the mixed image includes making, in a case where the reference position is set to the body part of the first users, position information of the body part of the first user and position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of the real video and the virtual object (“[0106] The augmentation system 208 computes the reference point to be any point that lies within a region corresponding to the real-world object. As an example, the reference point may be any one or combination of more than one skeletal joint position. [0112]….. The movement vector can be selected by the user from a user interface that displays icons representing different movement vectors. In another example, the movement vector can be received by a first user from a second user in a communication (e.g., a chat message). The movement vector stores changes in 3D positioning”).
Regarding claim 7 Assouline teaches wherein the setting includes being capable of changing the reference position (“[0106]….. Once the skeletal joint position or combination of multiple skeletal joint positions are selected, the augmentation system 208 uses their change in position throughout a video to adjust the reference point”).
Regarding claim 8 Assouline teaches wherein the receiving includes receiving of information concerning a position of the body part of the second user or information concerning the line of sight of the second user, as the motion information (“[0112]…..The skeletal joints in the movement vector are mapped to a skeletal rig of the virtual object to cause the virtual object to mimic movement of the corresponding skeletal joints stored in the movement vector. As an example, if the movement vector indicates that the arm joints moved up from one 3D coordinate to a second 3D coordinate at a certain distance and at a certain speed over the course of a given time interval (e.g., 3 seconds)”).
Regarding claim 11 Assouline teaches generate an avatar of the first user; and control motion of the avatar of the first user based on the motion information (“[0016]…… In order to dynamically adjust the placement, positioning and movement of the augmented reality item relative to the person in the scene, a set of skeletal joints of the person are identified and movement of such joints is stored in a movement vector. Subsequently, the movement vector is used to updated 3D movement of an augmented reality item, such as an avatar. [0039] The augmentation system 208 provides functionality to generate, display, and track virtual objects at positions relative to a real-world object (e.g., a person) depicted in a video captured by the client device 102. For example, the augmentation system 208 tracks virtual objects or augmented reality items (e.g., avatars) within at positions relative to real-world objects featured in a real-world scene of the video. [0118]….. Each avatar is presented simultaneously with the other avatars and is animated in accordance with the movement vector associated with the respective avatar. The user can tap on a given animated avatar to share with another user.”).
Regarding claim 12 Assouline teaches a display unit configured to display the mixed video generated by the generating of the mixed video (“[0038]….. The one or more images, videos, or augmented reality graphical elements are retrieved and presented as an overlay on top of the images that are captured and displayed by the front-facing camera of the client device 102”).
Regarding claim 13 Assouline teaches prevent, in a case where a body part of the first user is included in the mixed video to be displayed on the display unit, the body part of the first user from being displayed (In Fig. 11 part of the real-world object 1101 is not shown in graphical user interface 1110).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Assouline in view of BYEON et al. (US Pat. Pub. No. 20220191457 “Byeon”).
Regarding claim 6 Assouline is silent about wherein the generating of the mixed image includes making, in a case where the reference position is set to the attention object, position information of the attention object and the position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of the real video and the virtual object.
Byeon teaches making, in a case where reference position is set to attention object, position information of the attention object and the position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of multiple objects (“[0059]….the wearable display device 201 periodically transmits information on an image captured through a camera (not shown) of the wearable display device 201 and gaze information (e.g., a field of view (FOV) or an angle of view (AOV)) of a user, to the electronic device 101, and/or may transmit a change (e.g., a change in the location or the direction) in the state of the wearable display device 201 to the electronic device 101. [0060] According to various embodiments, the electronic device 101 may combine, process or arrange multiple pieces of graphic data and transmit the composite (as used herein, the term “composed” may refer to combining, processing or arranging multiple pieces of data”);
Byeon and Assouline are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline by making, in a case where the reference position is set to the attention object, position information of the attention object and the position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of the real video and the virtual object similar to making, in a case where reference position is set to attention object, position information of the attention object and the position information of the body part of the second user, which is included in the motion information, correspondent to each other, as position alignment of multiple objects as taught by Byeon.
The motivation for the above is to create an augmented image that will be easier to view by second user.
Claim(s) 9, 15, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Assouline in view of Orr (US Pat. Pub. No. 20200371738 “Orr”).
Regarding claim 9 Assouline is silent about the generating of the virtual object includes generating of a virtual object of an arm of the second user.
Orr teaches generating of the virtual object includes generating of a virtual object of an arm of the second user (“ [0030]….. Similarly, a human element (e.g., arm) can be substituted with an augmented reality graphic, such as an augmented reality arm or an augmented reality baseball bat. The augmented reality arm may be given visual effects such as being made translucent, turned black and white, or shown as another color”);
Orr and Assouline are analogous art as both of them are related to video processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline by generating of the virtual object includes generating of a virtual object of an arm of the second user as taught by Orr.
The motivation for the above is to enhance the applicability of Assouline by having different types of virtual object.
Regarding claim 15 Assouline is silent about wherein the video processing apparatus is a head mounted display.
Orr teaches video processing apparatus is a head mounted display (“[0036] In various embodiments, the VRTS 500 includes a third user 515 at a third location 508. In various embodiments, the third user may be wearing the VR/AR headset and/or motion tracking sensors as described in more detail with respect to FIG. 7. In various embodiments, the third user 515 may be an instructor, trainer, physical therapist, or other healthcare provider. In various embodiments, the third user 515 may transmit video to the first user 505 and the second user 510 via a camera 520 connected to a computer node implementing the VRTS 500. In various embodiments, the camera 520 may be a digital camera. In various embodiments, the camera 520 may be an omnidirectional camera”);
Orr and Assouline are analogous art as both of them are related to video processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline by having video processing apparatus as a head mounted display as taught by Orr.
The motivation for the above is to enhance applicability of Assouline by including different display device.
Regarding claim 16 Assouline teaches A video processing system including a first video processing apparatus that is used by a first user and processes a video; a second video processing apparatus that is used by a second user different from the first user and processes a video, (“[0015] Among other things, embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by rendering an augmented reality item and effects as if the augmented reality object exists in a real-world scene containing real-world objects featured in a video. [0018] FIG. 1 is a block diagram showing an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of a client device 102, each of which hosts a number of applications, including a messaging client 104 and other external applications 109 (e.g., third-party applications). [0178] The machine 1300 may include processors 1304, memory memory/storage 1306, and I/O components 1318, which may be configured to communicate with each other such as via a bus 1302”), and
wherein the first video processing apparatus comprises one or more processors and/or circuitry ([0018] FIG. 1 is a block diagram showing an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of a client device 102, each of which hosts a number of applications, including a messaging client 104 and other external applications 109 (e.g., third-party applications). [0178] The machine 1300 may include processors 1304, memory memory/storage 1306, and I/O components 1318, which may be configured to communicate with each other such as via a bus 1302”)configured to:
acquire a real video of a real space visually recognized by the first user (“[0039] The augmentation system 208 provides functionality to generate, display, and track virtual objects at positions relative to a real-world object (e.g., a person) depicted in a video captured by the client device 102”);
transmit the acquired real video to the second video processing apparatus used by the second user different from the first user (“[0019]…..The data exchanged between messaging clients 104, and between a messaging client 104 and the messaging server system 108, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video or other multimedia data)”);
receive motion information concerning motion of the second user from the second video processing apparatus(“[0129] At operation 701, the augmentation system 208 implemented by a client device 102 associated with a first user receives a communication from a second user. For example, a client device 102 of a second user transmits a chat message to a first user that attaches or includes a movement vector recorded by the second user.”);
generate a virtual object of a body part of the second user, which can be displayed in the video, based on the received motion information (“[0132] At operation 704, the augmentation system 208 animates the 3D avatar to mimic the 3D movement of the set of skeletal joints of the second user. [0133]…..In one example, the augmentation system 208 maintains a set of templates to generate virtual objects to be displayed in the video. Upon receiving a selection of a template from among the set of templates, and a selection of a position in the video, the augmentation system 208 generates and assigns the virtual object to the position within the three-dimensional space of the video”); and
generate a mixed video by mixing the real video and the virtual object, and wherein the generating of the mixed video includes generating of the mixed video by aligning respective positions of the real video and the virtual object(“[0153] At operation 804, the augmentation system 208 detects a 3D reference point of a real-world object depicted in a video (e.g., a real-time video feed being captured by a client device 102). For example, the augmentation system 208 selects one or more skeletal joints of a person depicted in the video and computes a 3D coordinate of the selected skeletal joints as the 3D reference point.
[0154] At operation 806, the augmentation system 208 orients the virtual object based on the 3D reference point”), and
wherein the second video processing apparatus comprises one or more processors and/or circuitry configured to: receive the transmitted real video ((“[0019]…..The data exchanged between messaging clients 104, and between a messaging client 104 and the messaging server system 108, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video or other multimedia data)”) but is silent about display the received real video;
Orr teaches receive transmitted real video and display the received real video (“[0036] In various embodiments, the VRTS 500 includes a third user 515 at a third location 508. In various embodiments, the third user may be wearing the VR/AR headset and/or motion tracking sensors as described in more detail with respect to FIG. 7. In various embodiments, the third user 515 may be an instructor, trainer, physical therapist, or other healthcare provider. In various embodiments, the third user 515 may transmit video to the first user 505 and the second user 510 via a camera 520 connected to a computer node implementing the VRTS 500. In various embodiments, the camera 520 may be a digital camera. In various embodiments, the camera 520 may be an omnidirectional camera”);
Orr and Assouline are analogous art as both of them are related to video processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline by receiving transmitted real video and displaying the received real video as taught by Orr.
The motivation for the above is that second user can provide some instruction based on the video.
Assouline modified by Orr teaches acquire motion information concerning motion of the second user; and transmit the acquired motion information to the first video processing apparatus (Assouline “[0129] At operation 701, the augmentation system 208 implemented by a client device 102 associated with a first user receives a communication from a second user. For example, a client device 102 of a second user transmits a chat message to a first user that attaches or includes a movement vector recorded by the second user.”).
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Assouline in view of Smith et al. (US Pat. No. 12101360 “Smith”).
Regarding claim 10 Assouline teaches be capable of performing an operation on the virtual object on the mixed video (“[0036] The augmentation system 208 provides various functions that enable a user to augment (e.g., annotate or otherwise modify or edit) media content associated with a message”) but is silent about set whether or not to permit the second user to perform the operation.
Smith teaches whether or not to permit second user to perform an operation (Col 10 lines 40-50 “As shown, the first user may specify configuration parameters such as identifiers that correspond to users who are permitted to join the virtual interaction session, whether the first user is to approve of a real-time communication link before it is established (“Call in” versus “Drop in”), a session duration, a micro-chat duration, and a blur level. The session duration defines a time limit of the virtual interaction session and the micro-chat duration defines a time limit for real-time communication links established during the virtual interaction session. There can be multiple micro-chat sessions within a virtual interactive session. A real-time communication link with a time limit may be referred to herein as a “time limited real-time communication link” or simply as “micro-chat.””);
Smith and Assouline are analogous art as both of them are related to video processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline to whether or not to permit second user to perform an operation as taught by Smith.
The motivation for the above is to provide the supervisory capability to the first user.
Claim(s) 14 is rejected under 35 U.S.C. 103 as being unpatentable over Assouline in view of Holmes (US Pat. Pub. No. 20230386147 “Holmes”).
Regarding claim 14 Assouline is silent is silent about eliminate, in a case where a body part of the first user is included in the mixed video to be displayed on the display unit, a difference in physiques between the first user and the second user in the body part of the first user and the body part of the second user, which is being displayed as the virtual object.
Holmes teaches eliminate, in a case where a body part of the first user is included in a mixed video to be displayed on the display unit, a difference in physiques between the first user and the second user in body part of the first user and the body part of the second user, which is being displayed as virtual object (“[0209]….. it may be useful to employ an auto-scaling function in which the size of the human element in each video source is scaled to more appropriately match the human element in the other video source. This may be especially important in examples in which contact is simulated between the human elements from each video source.
[0210] The size of the human element in the video may be dependent on the distance between the camera and the human element. Accordingly, the difference in proportions between the human elements may be most noticeable when one or more of the human elements being combined in the superimposed video is closer or further from the camera than expected. Accordingly, the system 10 may auto-scale, auto-zoom, or simply provide some indication to the user to adjust the position to make the human elements within the superimposed video appear more natural in terms of scale and proportion”);
Holmes and Assouline are analogous art as both of them are related to video processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Assouline by eliminating, in a case where a body part of the first user is included in a mixed video to be displayed on the display unit, a difference in physiques between the first user and the second user in body part of the first user and the body part of the second user, which is being displayed as virtual object as taught by Holmes.
The motivation for the above is to appear the body parts in more natural in terms of scale and proportion.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAPTARSHI MAZUMDER whose telephone number is (571)270-3454. The examiner can normally be reached 8 am-4 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571)272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAPTARSHI MAZUMDER/ Primary Examiner, Art Unit 2612
Prosecution Insights