DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on January 15, 2026 has been entered.
Response to Amendment
The amendment filed January 15, 2026 has been entered. Claims 1-3, 8-13, 18-20, 22-23, and 25-29 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Pages 13-15 of Remarks, filed January 15, 2026, with respect to the rejection(s) of claims 1, 11, and 20 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Ecovacs Deebot (Video 1 – DEEBOT T10 Plus_How to use [video-manager], Video 2 – ECOVACS DEEBOT T10 PLUS - Full Review - Is It the BEST?).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 11, 20, 23, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Kolen et al. (US 20200306640 A1) in view of Ecovacs Deebot (Video 1 - DEEBOT T10 Plus_How to use [video-manager], Video 2 - ECOVACS DEEBOT T10 PLUS - Full Review - Is It the BEST?) and Yamauchi et al. (JP 2019009752 A), hereinafter Kolen, Ecovacs, and Yamauchi respectively.
Regarding claim 1, Kolen teaches a server (Paragraph 0029 – “the interactive computing system 120 may be associated with a network-based service, which may be operated by a game publisher, game developer, platform provider or other entity”; Note: the interactive computing system is a server) comprising:
memory storing instructions (Paragraph 0118 – “Program code can be stored in ROM 46, RAM 48 or storage 40”; Note: program code is instructions);
and at least one processor configured to execute the instructions to (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”):
obtain, from an electronic device, first image information including at least one first image including a first area of an object taken by the electronic device (Fig. 3, Paragraph 0067-0068 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character. The input media may be provided by a player, such as by the player selecting to upload images or videos of the player or another person to be made into a virtual character from the player computing system 102 (or another computing device of the player's) to the interactive computing system 120… The images and/or video obtained at block 202 may have been originally captured by a camera of the player using a traditional two-dimensional (2D) camera”; Note: the interactive computing system obtains image data, which is equivalent to the first image information, from the player computing device. Fig. 3 shows the first image (input media 302) including a first area of the player, which is the head of the player; see screenshot of Fig. 3 below),
PNG
media_image1.png
441
549
media_image1.png
Greyscale
Screenshot of Fig. 3 (taken from Kolen)
obtain second image information for the object (Paragraph 0067 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character…additionally, the player may enable the interactive computing system 120 to access a third party media source 103, such as a social media account or social network account of the player, a cloud storage service used by the player to store digital media, home security camera or other video monitoring services, and/or other sources of media, as discussed above”; Note: the interactive computing system obtains third party media, which is equivalent to the second image information),
generate avatar information for displaying an avatar corresponding to the object in a virtual space of a metaverse (Paragraph 0091, 0100 – “the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world of one or more video games played by the player who requested creation of the custom virtual character…The custom virtual character data may be stored by the custom character system 130 in one or more formats that are used by one or more particular video games, such that the custom character may be rendered within the particular game's virtual environment”; Note: the avatar is incorporated/displayed into the virtual world, which is equivalent to the metaverse), based on the first image information and the second image information (Paragraph 0044, 0070 – “The appearance learning system 136 may be configured to analyze input media, such as images or videos depicting a person to be generated as a virtual character, to learn unique visual characteristics or appearance details of that person…the custom character system 130 may generate one or more custom behavior models and one or more custom appearance models from the input media. In some embodiments, a custom behavior model may be generated by the behavior learning system 134, and a custom appearance model (which may include custom 3D mesh data and corresponding textures) may be generated by the appearance learning system 136”; Note: the custom behavior model and appearance model are avatar information. The models are generated from the input media of the user),
and transmit the avatar information to a metaverse server for providing the virtual space of the metaverse (Paragraph 0032, 0057, 0091 – “the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which is equivalent to the avatar information, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server).
Kolen does not teach obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner in a plurality of directions around the object for 360- degree capturing of the object. However, Ecovacs teaches obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner (Video 1: Images 1-4 – The images 1-4 show the robot cleaner capturing a plurality of images in a plurality of directions) in a plurality of directions around the object for 360- degree capturing of the object (Video 2: Images 1-4 – The images 1-4 (timestamp 6:00-6:01) show the robot cleaner moving in a plurality of directions around an object in a 360-degree movement). It would have been obvious to combine the features of Ecovacs to have the robot cleaner perform 360-degree image capturing because the user may want to have a full view of an object or themselves, which could be achieved by taking images using the robot cleaner while it is moving around the object. This would be more convenient than using a regular camera, like from a phone, since it would require manually moving the phone around to different positions. Moreover, since the server in Kolen already receives image information from a third party media source, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Ecovacs to receive image information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves. Additionally, many robot cleaners already have cameras or sensors for capturing images and detecting objects, so it would be convenient to receive image information from a robot cleaner, just like any other device, such as a smartphone or security camera.
Kolen modified by Ecovacs still does not teach the plurality of second images including a second area of the object that is lower than the first area; nor that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap. However, Yamauchi teaches the plurality of second images including a second area of the object that is lower than the first area (Paragraph 0049, 0059 – “The video analysis unit 12 receives video signals from a plurality of cameras 11 (11a, 11b, . . . ) installed in various locations in a monitored area…As shown in Figure 3, entrance camera 11e has a longer focal length and a narrower viewing angle than cameras 11a and 11b, which are positioned to monitor and understand the entire venue, and its installation height and position are determined so that the entire body (head, upper body, lower body) including the face of a person HM walking through entrance ET is included within the field of view”; Note: the video signals are image information. The second image captures the entire body of the person, which is the second area and captures lower than a first area); and that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap (Fig. 3, Paragraph 0007-0008 – “personal characteristic information and coordinate information are extracted based on images from a plurality of cameras that capture the area to be monitored at different angles so that there is some overlap in the effective capture range…The avatar generation unit can be configured, for example, to generate an avatar image of a person by integrating person feature information extracted individually for the same person from video footage captured by multiple cameras by the person feature information extraction unit”; Note: as shown in the modified screenshot of Fig. 3, there is an overlap between the first and second area).
PNG
media_image2.png
498
861
media_image2.png
Greyscale
Modified screenshot of Fig. 3 (taken from Yamauchi)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to have one portion be lower than the other for the benefit of being able to accurately capture a representation of a tall object, like a person. If the field of view of one camera is not large enough to capture the whole object, taking an image at a high angle and then a low angle would ensure that the full length of the object is captured. Additionally, the robot cleaner of Ecovacs takes images from a low angle, as shown in Video 1: Image 3, so logically, the robot cleaner’s images would likely show a lower area than the camera’s images from Kolen, which were intended to properly capture a player. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to generate avatar information corresponding to an overlapping region for the benefit of “improving the realism (resolution) of avatar images, improving positional accuracy when combining avatar images with background images, and even improving the accuracy of determining whether people captured on camera are similar or different” (Yamauchi: Paragraph 0007). In other words, being able to combine images that capture different areas of an object allows for better quality avatars that correspond to the object. Including the overlap between the areas will further ensure that all the parts of the object are represented.
Regarding claim 11, Kolen teaches a method (Paragraph 0066 – “the method 200, in whole or in part, can be implemented by a game application 104, a custom character system 130, a player computing system 102, an interactive computing system 120, or other application component or module”) performed by a server (Paragraph 0029 – “the interactive computing system 120 may be associated with a network-based service, which may be operated by a game publisher, game developer, platform provider or other entity”; Note: the interactive computing system is a server), comprising:
obtaining, from an electronic device, first image information including at least one first image including a first area of an object taken by the electronic device (Fig. 3, Paragraph 0067-0068 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character. The input media may be provided by a player, such as by the player selecting to upload images or videos of the player or another person to be made into a virtual character from the player computing system 102 (or another computing device of the player's) to the interactive computing system 120… The images and/or video obtained at block 202 may have been originally captured by a camera of the player using a traditional two-dimensional (2D) camera”; Note: the interactive computing system obtains image data, which is equivalent to the first image information, from the player computing device. Fig. 3 shows the first image (input media 302) including a first area of the player, which is the head of the player; see screenshot of Fig. 3 above),
obtaining second image information for the object (Paragraph 0067 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character…additionally, the player may enable the interactive computing system 120 to access a third party media source 103, such as a social media account or social network account of the player, a cloud storage service used by the player to store digital media, home security camera or other video monitoring services, and/or other sources of media, as discussed above”; Note: the interactive computing system obtains third party media, which is equivalent to the second image information),
generating avatar information for displaying an avatar corresponding to the object in a virtual space of a metaverse (Paragraph 0091, 0100 – “the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world of one or more video games played by the player who requested creation of the custom virtual character…The custom virtual character data may be stored by the custom character system 130 in one or more formats that are used by one or more particular video games, such that the custom character may be rendered within the particular game's virtual environment”; Note: the avatar is incorporated/displayed into the virtual world, which is equivalent to the metaverse), based on the first image information and the second image information (Paragraph 0044, 0070 – “The appearance learning system 136 may be configured to analyze input media, such as images or videos depicting a person to be generated as a virtual character, to learn unique visual characteristics or appearance details of that person…the custom character system 130 may generate one or more custom behavior models and one or more custom appearance models from the input media. In some embodiments, a custom behavior model may be generated by the behavior learning system 134, and a custom appearance model (which may include custom 3D mesh data and corresponding textures) may be generated by the appearance learning system 136”; Note: the custom behavior model and appearance model are avatar information. The models are generated from the input media of the user),
and transmitting the avatar information to a metaverse server for providing the virtual space of the metaverse (Paragraph 0032, 0057, 0091 – “the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which is equivalent to the avatar information, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server).
Kolen does not teach obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner in a plurality of directions around the object for 360- degree capturing of the object. However, Ecovacs teaches obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner (Video 1: Images 1-4 – The images 1-4 show the robot cleaner capturing a plurality of images in a plurality of directions) in a plurality of directions around the object for 360- degree capturing of the object (Video 2: Images 1-4 – The images 1-4 (timestamp 6:00-6:01) show the robot cleaner moving in a plurality of directions around an object in a 360-degree movement). It would have been obvious to combine the features of Ecovacs to have the robot cleaner perform 360-degree image capturing because the user may want to have a full view of an object or themselves, which could be achieved by taking images using the robot cleaner while it is moving around the object. This would be more convenient than using a regular camera, like from a phone, since it would require manually moving the phone around to different positions. Moreover, since the server in Kolen already receives image information from a third party media source, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Ecovacs to receive image information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves. Additionally, many robot cleaners already have cameras or sensors for capturing images and detecting objects, so it would be convenient to receive image information from a robot cleaner, just like any other device, such as a smartphone or security camera.
Kolen modified by Ecovacs still does not teach the plurality of second images including a second area of the object; nor that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap. However, Yamauchi teaches the plurality of second images including a second area of the object (Paragraph 0049, 0059 – “The video analysis unit 12 receives video signals from a plurality of cameras 11 (11a, 11b, . . . ) installed in various locations in a monitored area…As shown in Figure 3, entrance camera 11e has a longer focal length and a narrower viewing angle than cameras 11a and 11b, which are positioned to monitor and understand the entire venue, and its installation height and position are determined so that the entire body (head, upper body, lower body) including the face of a person HM walking through entrance ET is included within the field of view”; Note: the video signals are image information. The second image captures the entire body of the person, which is the second area and captures lower than a first area); and that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap (Fig. 3, Paragraph 0007-0008 – “personal characteristic information and coordinate information are extracted based on images from a plurality of cameras that capture the area to be monitored at different angles so that there is some overlap in the effective capture range…The avatar generation unit can be configured, for example, to generate an avatar image of a person by integrating person feature information extracted individually for the same person from video footage captured by multiple cameras by the person feature information extraction unit”; Note: as shown in the modified screenshot of Fig. 3, there is an overlap between the first and second area). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to have one portion be lower than the other for the benefit of being able to accurately capture a representation of a tall object, like a person. If the field of view of one camera is not large enough to capture the whole object, taking an image at a high angle and then a low angle would ensure that the full length of the object is captured. Additionally, the robot cleaner of Ecovacs takes images from a low angle, as shown in Video 1: Image 3, so logically, the robot cleaner’s images would likely show a lower area than the camera’s images from Kolen, which were intended to properly capture a player. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to generate avatar information corresponding to an overlapping region for the benefit of “improving the realism (resolution) of avatar images, improving positional accuracy when combining avatar images with background images, and even improving the accuracy of determining whether people captured on camera are similar or different” (Yamauchi: Paragraph 0007). In other words, being able to combine images that capture different areas of an object allows for better quality avatars that correspond to the object. Including the overlap between the areas will further ensure that all the parts of the object are represented.
Regarding claim 20, Kolen teaches a non-transitory storage medium storing instructions that, when executed by at least one processor (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors. The code modules may be stored in any type of non-transitory computer-readable medium or other computer storage device”) of a server (Paragraph 0029 – “the interactive computing system 120 may be associated with a network-based service, which may be operated by a game publisher, game developer, platform provider or other entity”; Note: the interactive computing system is a server), cause the server to:
obtain, from an electronic device, first image information including at least one first image including a first area of an object taken by the electronic device (Fig. 3, Paragraph 0067-0068 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character. The input media may be provided by a player, such as by the player selecting to upload images or videos of the player or another person to be made into a virtual character from the player computing system 102 (or another computing device of the player's) to the interactive computing system 120… The images and/or video obtained at block 202 may have been originally captured by a camera of the player using a traditional two-dimensional (2D) camera”; Note: the interactive computing system obtains image data, which is equivalent to the first image information, from the player computing device. Fig. 3 shows the first image (input media 302) including a first area of the player, which is the head of the player; see screenshot of Fig. 3 above),
obtain second image information for the object (Paragraph 0067 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character…additionally, the player may enable the interactive computing system 120 to access a third party media source 103, such as a social media account or social network account of the player, a cloud storage service used by the player to store digital media, home security camera or other video monitoring services, and/or other sources of media, as discussed above”; Note: the interactive computing system obtains third party media, which is equivalent to the second image information),
generate avatar information for displaying an avatar corresponding to the object in a virtual space of a metaverse (Paragraph 0091, 0100 – “the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world of one or more video games played by the player who requested creation of the custom virtual character…The custom virtual character data may be stored by the custom character system 130 in one or more formats that are used by one or more particular video games, such that the custom character may be rendered within the particular game's virtual environment”; Note: the avatar is incorporated/displayed into the virtual world, which is equivalent to the metaverse), based on the first image information and the second image information (Paragraph 0044, 0070 – “The appearance learning system 136 may be configured to analyze input media, such as images or videos depicting a person to be generated as a virtual character, to learn unique visual characteristics or appearance details of that person…the custom character system 130 may generate one or more custom behavior models and one or more custom appearance models from the input media. In some embodiments, a custom behavior model may be generated by the behavior learning system 134, and a custom appearance model (which may include custom 3D mesh data and corresponding textures) may be generated by the appearance learning system 136”; Note: the custom behavior model and appearance model are avatar information. The models are generated from the input media of the user),
and transmit the avatar information to a metaverse server for providing the virtual space of the metaverse (Paragraph 0032, 0057, 0091 – “the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which is equivalent to the avatar information, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server).
Kolen does not teach obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner in a plurality of directions around the object for 360- degree capturing of the object. However, Ecovacs teaches obtaining, from a robot cleaner, second image information including a plurality of second images taken by the robot cleaner (Video 1: Images 1-4 – The images 1-4 show the robot cleaner capturing a plurality of images in a plurality of directions) in a plurality of directions around the object for 360- degree capturing of the object (Video 2: Images 1-4 – The images 1-4 (timestamp 6:00-6:01) show the robot cleaner moving in a plurality of directions around an object in a 360-degree movement). It would have been obvious to combine the features of Ecovacs to have the robot cleaner perform 360-degree image capturing because the user may want to have a full view of an object or themselves, which could be achieved by taking images using the robot cleaner while it is moving around the object. This would be more convenient than using a regular camera, like from a phone, since it would require manually moving the phone around to different positions. Moreover, since the server in Kolen already receives image information from a third party media source, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Ecovacs to receive image information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves. Additionally, many robot cleaners already have cameras or sensors for capturing images and detecting objects, so it would be convenient to receive image information from a robot cleaner, just like any other device, such as a smartphone or security camera.
Kolen modified by Ecovacs still does not teach the plurality of second images including a second area of the object lower than the first area; nor that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap. However, Yamauchi teaches the plurality of second images including a second area of the object lower than the first area (Paragraph 0049, 0059 – “The video analysis unit 12 receives video signals from a plurality of cameras 11 (11a, 11b, . . . ) installed in various locations in a monitored area…As shown in Figure 3, entrance camera 11e has a longer focal length and a narrower viewing angle than cameras 11a and 11b, which are positioned to monitor and understand the entire venue, and its installation height and position are determined so that the entire body (head, upper body, lower body) including the face of a person HM walking through entrance ET is included within the field of view”; Note: the video signals are image information. The second image captures the entire body of the person, which is the second area and captures lower than a first area); and that the avatar information includes rendering information of a third area of the object where the first area and the second area overlap (Fig. 3, Paragraph 0007-0008 – “personal characteristic information and coordinate information are extracted based on images from a plurality of cameras that capture the area to be monitored at different angles so that there is some overlap in the effective capture range…The avatar generation unit can be configured, for example, to generate an avatar image of a person by integrating person feature information extracted individually for the same person from video footage captured by multiple cameras by the person feature information extraction unit”; Note: as shown in the modified screenshot of Fig. 3, there is an overlap between the first and second area). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to have one portion be lower than the other for the benefit of being able to accurately capture a representation of a tall object, like a person. If the field of view of one camera is not large enough to capture the whole object, taking an image at a high angle and then a low angle would ensure that the full length of the object is captured. Additionally, the robot cleaner of Ecovacs takes images from a low angle, as shown in Video 1: Image 3, so logically, the robot cleaner’s images would likely show a lower area than the camera’s images from Kolen, which were intended to properly capture a player. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to generate avatar information corresponding to an overlapping region for the benefit of “improving the realism (resolution) of avatar images, improving positional accuracy when combining avatar images with background images, and even improving the accuracy of determining whether people captured on camera are similar or different” (Yamauchi: Paragraph 0007). In other words, being able to combine images that capture different areas of an object allows for better quality avatars that correspond to the object. Including the overlap between the areas will further ensure that all the parts of the object are represented.
Regarding claim 23, Kolen in view of Ecovacs and Yamauchi teaches the server of claim 1. Kolen further teaches wherein the avatar information includes first avatar information representing a portion of the avatar corresponding to a head portion of the object (Paragraph 0078, 0085 – “The illustrative method 400 begins at block 402, where the custom character system 130 may obtain input visual media depicting a real person, as well as generic model data…the generic model data may include 3D mesh data for separate body parts, facial features, hair and/or other elements as modular components. In other embodiments, the 3D mesh data may incorporate all body and face features within a comprehensive base 3D mesh….blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: avatar information includes information representing a face, which corresponds to a head portion of a person), and wherein the avatar information includes second avatar information representing a portion of the avatar corresponding to a body portion of the object (Paragraph 0078, 0085 – “The illustrative method 400 begins at block 402, where the custom character system 130 may obtain input visual media depicting a real person, as well as generic model data…the generic model data may include 3D mesh data for separate body parts, facial features, hair and/or other elements as modular components. In other embodiments, the 3D mesh data may incorporate all body and face features within a comprehensive base 3D mesh…blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: avatar information includes information representing a body of a person).
Regarding claim 26, Kolen in view of Ecovacs and Yamauchi teaches the method of claim 11. Kolen further teaches wherein the avatar information includes first avatar information representing a portion of the avatar corresponding to a head portion of the object (Paragraph 0078, 0085 – “The illustrative method 400 begins at block 402, where the custom character system 130 may obtain input visual media depicting a real person, as well as generic model data…the generic model data may include 3D mesh data for separate body parts, facial features, hair and/or other elements as modular components. In other embodiments, the 3D mesh data may incorporate all body and face features within a comprehensive base 3D mesh….blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: avatar information includes information representing a face, which corresponds to a head portion of a person), and wherein the avatar information includes second avatar information representing a portion of the avatar corresponding to a body portion of the object (Paragraph 0078, 0085 – “The illustrative method 400 begins at block 402, where the custom character system 130 may obtain input visual media depicting a real person, as well as generic model data…the generic model data may include 3D mesh data for separate body parts, facial features, hair and/or other elements as modular components. In other embodiments, the 3D mesh data may incorporate all body and face features within a comprehensive base 3D mesh…blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: avatar information includes information representing a body of a person).
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Kolen in view of Ecovacs, Yamauchi, and Seo et al. (US 20160278599 A1), hereinafter Seo.
Regarding claim 2, Kolen in view of Ecovacs and Yamauchi teaches the server of claim 1. Kolen further teaches wherein the at least one processor is further configured to execute the instructions (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”) to: receive action information of the object (Paragraph 0094 – “the behavior learning system 134 may extract or determine traits and typical action sequences of the person depicted in the input media based on an automated analysis of the input media”), generate update information for updating the avatar information based on the action information (Paragraph 0096 – “the behavior learning system 134 may generate one or more custom character behavior models based at least in part on differences between the generic behavior model(s) obtained at block 502 and behavior data extracted or determined from the input media analysis at block 504. For example, the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media…the behavior learning system 134 may alter the propensities for the customized virtual character to engage in particular actions in particular orders when there is significant deviation from the standard behaviors and the particular observed person's behaviors”; Note: the custom character behavior model is equivalent to the update information, as it contains information about the differences between the default model and the behavioral data), and transmit the update information to the metaverse server (Paragraph 0005, 0032, 0057, 0091 – “custom virtual character data corresponding to the real person, wherein the custom virtual character data includes at least the custom 3D model data and the custom behavior data… the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which includes the behavior/update information, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server). Kolen does not teach receiving action information of the object from the robot cleaner. However, Seo teaches receiving action information of the object from the robot cleaner (Paragraph 0198 – “The object tracking unit 440 may execute tracking of a verified object. For example, an object in sequentially acquired stereo images may be verified, motion or a motion vector of the verified object may be calculated, and movement of the corresponding object may be tracked based on the calculated motion or motion vector. Thereby, the object tracking unit 440 may track a person, an animal, an object, and/or the like, located proximate the robot cleaner 200”; Note: the robot cleaner collects motion/action information of an object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Seo to receive action information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves and objects around the house. Additionally, many robot cleaners already have sensors for detecting objects and motion for cleaning purposes, so it would be convenient to receive action information from a robot cleaner.
Regarding claim 12, Kolen in view of Ecovacs and Yamauchi teaches the method of claim 11. Kolen further teaches receiving action information of the object (Paragraph 0094 – “the behavior learning system 134 may extract or determine traits and typical action sequences of the person depicted in the input media based on an automated analysis of the input media”), generating update information for updating the avatar information based on the action information (Paragraph 0096 – “the behavior learning system 134 may generate one or more custom character behavior models based at least in part on differences between the generic behavior model(s) obtained at block 502 and behavior data extracted or determined from the input media analysis at block 504. For example, the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media…the behavior learning system 134 may alter the propensities for the customized virtual character to engage in particular actions in particular orders when there is significant deviation from the standard behaviors and the particular observed person's behaviors”; Note: the custom character behavior model is equivalent to the update information, as it contains information about the differences between the default model and the behavioral data), and transmitting the update information to the metaverse server (Paragraph 0005, 0032, 0057, 0091 – “custom virtual character data corresponding to the real person, wherein the custom virtual character data includes at least the custom 3D model data and the custom behavior data… the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which includes the behavior/update information, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server). Kolen does not teach receiving action information of the object from the robot cleaner. However, Seo teaches receiving action information of the object from the robot cleaner (Paragraph 0198 – “The object tracking unit 440 may execute tracking of a verified object. For example, an object in sequentially acquired stereo images may be verified, motion or a motion vector of the verified object may be calculated, and movement of the corresponding object may be tracked based on the calculated motion or motion vector. Thereby, the object tracking unit 440 may track a person, an animal, an object, and/or the like, located proximate the robot cleaner 200”; Note: the robot cleaner collects motion/action information of an object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Seo to receive action information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves and objects around the house. Additionally, many robot cleaners already have sensors for detecting objects and motion for cleaning purposes, so it would be convenient to receive action information from a robot cleaner.
Claims 3, 13, and 28-29 are rejected under 35 U.S.C. 103 as being unpatentable over Kolen in view of Ecovacs, Yamauchi, and Chen et al. (US 20190035149 A1), hereinafter Chen.
Regarding claim 3, Kolen in view of Ecovacs and Yamauchi teaches the server of claim 1. Kolen further teaches generating an avatar mesh corresponding to an appearance of the object (Paragraph 0075 – “The input media 302 is analyzed by the custom character system 130 at block 306 to generate custom appearance data (such as one or more custom texture images) and an altered 3D mesh 308. The altered or custom 3D mesh 308 may be created by changing parameters of the generic 3D mesh data 304 based on the appearance of the person depicted in the input media”), generating texture information for the object based on the first image information and the second image information (Paragraph 0080 – “A GAN or cGAN applied at block 406 may take a generic visual model and/or texture (such as for edge extraction) and generate a full texture and/or other visual style elements that closely resemble the real person depicted in one or more input images and/or input video frames”; Note: texture information is generated based on the input images, which includes the first and second image information), and generating the avatar information by applying the texture information to the avatar mesh (Paragraph 0082 – “the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402)”; Note: applying the texture information to the mesh results in a 3D model, which is the avatar information). Kolen does not teach obtaining prior information of the object nor generating an avatar mesh corresponding to an appearance of the object based on the prior information. However, Chen teaches obtaining prior information of the object (Paragraph 0026, 0102, 0348 – “The method may be one in which using a shape prior selection process is used to find the most suitable shape prior from a library, using selection criteria such as the user's ethnicity, gender, age, and other attributes… The method may be one in which, in a first stage, a user can generate an initial 3D body avatar from the input of their body measurements through regression... The current regression-based approach gives an average modelling error of 2-3 cm over major body measurements when 6 input measurements (i.e. height, weight, cup size, underbust, waist, and hips) are provided”; Note: information of the user, like body measurements, is equivalent to the prior information) and generating an avatar corresponding to an appearance of the object based on the prior information (Paragraph 0026, 0102, 0348 – “The method may be one in which using a shape prior selection process is used to find the most suitable shape prior from a library, using selection criteria such as the user's ethnicity, gender, age, and other attributes… The method may be one in which, in a first stage, a user can generate an initial 3D body avatar from the input of their body measurements through regression... The current regression-based approach gives an average modelling error of 2-3 cm over major body measurements when 6 input measurements (i.e. height, weight, cup size, underbust, waist, and hips) are provided”; Note: an avatar is generated using the body measurements, which is appearance). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Chen to have prior information on the object and generate an avatar based on that information for the benefit of allowing the user to customize the avatar to their preferences, which enhances the user experience. It may also allow the user to ensure that the avatar is accurate to their body shape and features.
Regarding claim 13, Kolen in view of Ecovacs and Yamauchi teaches the method of claim 11. Kolen further teaches generating an avatar mesh corresponding to an appearance of the object (Paragraph 0075 – “The input media 302 is analyzed by the custom character system 130 at block 306 to generate custom appearance data (such as one or more custom texture images) and an altered 3D mesh 308. The altered or custom 3D mesh 308 may be created by changing parameters of the generic 3D mesh data 304 based on the appearance of the person depicted in the input media”), generating texture information for the object based on the first image information and the second image information (Paragraph 0080 – “A GAN or cGAN applied at block 406 may take a generic visual model and/or texture (such as for edge extraction) and generate a full texture and/or other visual style elements that closely resemble the real person depicted in one or more input images and/or input video frames”; Note: texture information is generated based on the input images, which includes the first and second image information), and generating the avatar information by applying the texture information to the avatar mesh (Paragraph 0082 – “the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402)”; Note: applying the texture information to the mesh results in a 3D model, which is the avatar information). Kolen does not teach obtaining prior information of the object nor generating an avatar mesh corresponding to an appearance of the object based on the prior information. However, Chen teaches obtaining prior information of the object (Paragraph 0026, 0102, 0348 – “The method may be one in which using a shape prior selection process is used to find the most suitable shape prior from a library, using selection criteria such as the user's ethnicity, gender, age, and other attributes… The method may be one in which, in a first stage, a user can generate an initial 3D body avatar from the input of their body measurements through regression... The current regression-based approach gives an average modelling error of 2-3 cm over major body measurements when 6 input measurements (i.e. height, weight, cup size, underbust, waist, and hips) are provided”; Note: information of the user, like body measurements, is equivalent to the prior information) and generating an avatar corresponding to an appearance of the object based on the prior information (Paragraph 0026, 0102, 0348 – “The method may be one in which using a shape prior selection process is used to find the most suitable shape prior from a library, using selection criteria such as the user's ethnicity, gender, age, and other attributes… The method may be one in which, in a first stage, a user can generate an initial 3D body avatar from the input of their body measurements through regression... The current regression-based approach gives an average modelling error of 2-3 cm over major body measurements when 6 input measurements (i.e. height, weight, cup size, underbust, waist, and hips) are provided”; Note: an avatar is generated using the body measurements, which is appearance). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Chen to have prior information on the object and generate an avatar based on that information for the benefit of allowing the user to customize the avatar to their preferences, which enhances the user experience. It may also allow the user to ensure that the avatar is accurate to their body shape and features.
Regarding claim 28, Kolen in view of Ecovacs, Yamauchi, and Chen teaches the server of claim 3. Kolen further teaches wherein the object is a user of the electronic device or a pet of the user (Paragraph 0067-0068 – “the media processing system 132 may obtain video data, image data and/or other input media depicting a person to be imported into a game or otherwise made into a virtual character. The input media may be provided by a player, such as by the player selecting to upload images or videos of the player… The player may do so via an in-game user interface generated by a game application…The images and/or video obtained at block 202 may have been originally captured by a camera of the player using a traditional two-dimensional (2D) camera”; Note: the object is the user/player). Kolen does not teach that the prior information includes height information of the user, weight information of the user, gender information of the user, and appearance information of the user in a case that the object is the user, and the prior information includes weight information of the pet, gender information of the pet, name information of the pet, and breed information of the pet in a case that the object is the pet. However, Chen teaches the prior information includes height information of the user, weight information of the user, gender information of the user, and appearance information of the user in a case that the object is the user (Paragraph 0026, 0102, 0348 – “The method may be one in which using a shape prior selection process is used to find the most suitable shape prior from a library, using selection criteria such as the user's ethnicity, gender, age, and other attributes… The method may be one in which, in a first stage, a user can generate an initial 3D body avatar from the input of their body measurements through regression... The current regression-based approach gives an average modelling error of 2-3 cm over major body measurements when 6 input measurements (i.e. height, weight, cup size, underbust, waist, and hips) are provided”; Note: information of the user includes body measurements like height and weight, gender, and appearance like ethnicity). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Chen to have the prior information include height, weight, gender, and appearance for the benefit of “creating photo-realistic visualisation of the users' 3D avatar” (Chen: Paragraph 0186). “The more closely the avatar resembles the user, the more compelling the user may find the technology and the more they may trust the technology” (Chen: Paragraph 0002).
Regarding claim 29, Kolen in view of Ecovacs, Yamauchi, and Chen teaches the server of claim 28. Kolen further teaches wherein the texture information for the object includes head texture information and body texture information (Paragraph 0080-0082, 0085 – “At block the 406, the appearance learning system 136 may provide input images and/or video data of the input media that depicts the real person as input to the retrieved visual style extraction model…A cGAN employed at block 406, in some embodiments, may include a generator that generates a visual style (which may include generating one or more images for application as textures to a 3D mesh or other 3D model)… At block the 408, the appearance learning system 136 may apply the extracted visual style that was generated or output by the model in block 406 as a custom texture on the generic 3D model…blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: there is visual style information, which is equivalent to the texture information. It includes visual style information for the head/face and the body), and the first image information is for generating the head texture information and the second image information is for generating the body texture information (Paragraph 0080-0082, 0085 – “At block the 406, the appearance learning system 136 may provide input images and/or video data of the input media that depicts the real person as input to the retrieved visual style extraction model…A cGAN employed at block 406, in some embodiments, may include a generator that generates a visual style (which may include generating one or more images for application as textures to a 3D mesh or other 3D model)… At block the 408, the appearance learning system 136 may apply the extracted visual style that was generated or output by the model in block 406 as a custom texture on the generic 3D model…blocks 404, 406, 408 and/or 410 may be performed separately for altering the virtual body to match the body of the real person and then again for altering the virtual face to match the face of the real person”; Note: there is visual style information, which is equivalent to the texture information. It includes visual style information for the head/face and the body. Since generating the texture occurs separately for the face and body, it would have been obvious to one of ordinary skill in the art to have head/face input images (first image information) be used for generating face texture and to have body input images (second image information) be used for generating body texture for the benefit of having the face texture accurately match the person’s face in the image and having the body texture accurately match the person’s body in the image). Kolen does not teach the first area of the object represents a head portion of the object and the second area of the object represents a body portion of the object. However, Yamauchi teaches the first area of the object represents a head portion of the object, the second area of the object represents a body portion of the object (Paragraph 0059 – “As shown in Figure 3, entrance camera 11e has a longer focal length and a narrower viewing angle than cameras 11a and 11b, which are positioned to monitor and understand the entire venue, and its installation height and position are determined so that the entire body (head, upper body, lower body) including the face of a person HM walking through entrance ET is included within the field of view. On the other hand, cameras 11a and 11b are mounted at a position slightly higher than entrance camera 11e”; Note: the first area is the head, and the second area is the body; see modified screenshot of Fig. 3 above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Yamauchi to have the first area represent a head and the second area represent a body because in Kolen, the textures are generated separately for the head and body, and therefore, having separate images for a first area of a head and second area of a body would make it easier to generate the separate textures. Additionally, not all cameras may be able to capture a full-body image (including the head), so having separate areas for the head and body is more convenient for achieving a complete representation of a person.
Claims 8-10, 18-19, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Kolen in view of Ecovacs, Yamauchi, Seo, and Gomez et al. (WO 2018178461 A1), hereinafter Gomez.
Regarding claim 8, Kolen in view of Ecovacs and Yamauchi teaches the server of claim 1. Kolen further teaches wherein the at least one processor is further configured to execute the instructions (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”) to: obtain object action information for the object (Paragraph 0094 – “the behavior learning system 134 may extract or determine traits and typical action sequences of the person depicted in the input media based on an automated analysis of the input media”); generate an observed action pattern of the object by performing a rule analysis based on the object action information (Paragraph 0095 – “the behavior learning system 134 may analyze a long video file or stream of video data (such as many minutes or many hours), label time stamps or frames in the video in which particular behaviors or actions are identified, link successive behaviors or actions as action sequences (e.g., sleep, then stand up, yawn, then walk with a limp briefly before walking more easily, etc.), then compile statistics of the frequency and length of the various observations in order to apply a Markov analysis or model to predict behavior of the person”); and generate an action tree of the object based on the observed action pattern (Paragraph 0096 – “the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media, which may be in the form of behavior trees, action sequences, and/or other formats”). Kolen does not teach obtaining object action information for the object from the robot cleaner; obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server; nor generating an observed action pattern of the object by performing a rule analysis based on the object action information and the IoT information. However, Seo teaches obtaining object action information for the object from the robot cleaner (Paragraph 0060 – “the cleaning robot 100 can grasp the position or movement of an animal based on the received direction when the sound of the animal is received through the microphone 265”; Note: the robot cleaner can collect information on movement of an animal, which is equivalent to object action information). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Seo to receive action information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves and objects around the house. Additionally, many robot cleaners already have sensors for detecting objects and motion, so it would be convenient to receive action information from a robot cleaner. Kolen modified by Seo still does not teach obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server; nor generating an observed action pattern of the object by performing a rule analysis based on the object action information and the IoT information. However, Gomez teaches obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server (Paragraph 0017, 0035-0037 – “The devices used can be Internet of Things (IoT) devices…The method also includes the following steps performed by the server: Collect information received from one or more of the monitoring devices of the group of living beings; Collect information received from one or more external data sources”); and generating an observed action pattern of the object based on the object action information and the IoT information (Paragraph 0038 – “Generate behavioral patterns for each monitored device (i.e., monitoring device) based on statistical processing of the collected information”; Note: the monitored device is an IoT device). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to obtain IoT information and use it to generate an action pattern because being connected to the Internet makes it easy for IoT devices to track data that changes overtime, which can include behaviors. This makes IoT devices a useful tool in monitoring users, and the data it collects can be utilized for other purposes, like managing health or even in a metaverse.
Regarding claim 9, Kolen in view of Ecovacs, Yamauchi, Seo, and Gomez teaches the server of claim 8. Kolen further teaches wherein the at least one processor is, to generate the observed action pattern, configured to execute the instructions (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”) to: generate condition information (Paragraph 00096-0097 – “the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time. The behavior learning system 134 may incorporate these statistics or likelihoods of certain actions into a stored behavior model…individual behaviors or actions in the behavior model may be linked or associated with environmental stimuli or objects. For example, the behavior data may indicate that the particular person is very likely to give other people a high five when seeing them for the first time in a video”; Note: in the examples given, standing up from a seated position and seeing someone for the first time are condition information), identify a specific action based on the object action information (Paragraph 0096 – “the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media…For example, the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time”; Note: in the example, a specific action (stretching) is identified based on the object action information), and generate the observed action pattern by associating the condition information and a specific action (Paragraph 0096 – “the behavior learning system 134 may generate one or more custom character behavior models based at least in part on differences between the generic behavior model(s) obtained at block 502 and behavior data extracted or determined from the input media analysis at block 504. For example, the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media, which may be in the form of behavior trees, action sequences, and/or other formats. For example, the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time. The behavior learning system 134 may incorporate these statistics or likelihoods of certain actions into a stored behavior model, such as a behavior tree”; Note: the custom character behavior model is equivalent to the observed action pattern. Conditions and actions are associated. For example, the condition of standing up is associated with the action of stretching). Kolen does not teach generating condition information based on time information and the IoT information nor identifying a specific action having a ratio greater than or equal to a threshold value in the condition information, based on the object action information. However, Gomez teaches generating condition information based on time information and the IoT information (Paragraph 0030-0031 – “a) Receive information on one or more physical, physiological and/or environmental variables of the living being carrying the monitoring device, measured by one or more sensors associated with the monitoring device; b) Obtain statistical parameters for each variable in time windows by processing the information received;”; Note: the time windows and the associated monitoring device are the condition information) and identifying a specific action having a ratio greater than or equal to a threshold value in the condition information, based on the object action information (Paragraph 0030-0033 – “a) Receive information on one or more physical, physiological and/or environmental variables of the living being carrying the monitoring device, measured by one or more sensors associated with the monitoring device; b) Obtain statistical parameters for each variable in time windows by processing the information received; c) Compare the value of each statistical parameter in the current time window with thresholds previously calculated by the monitoring device, based at least on previous values of said statistical parameters; d) If as a result of such comparison, no anomaly is detected in any of the statistical parameters, recalculate the value of each of the thresholds taking into account the current value of each statistical parameter in the current time window”; Note: the variable of the living being is equivalent to the action, and the specific variable can be identified to have a statistical value and is compared with other previous values, as a threshold, to evaluate its normality). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to generate conditions based on IoT and time information because observing IoT usage and time allows the action pattern to reflect the object/user’s schedule and track their common behaviors, which would then allow for an accurate representation of the object/user. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to identify a specific action exceeding a threshold of the condition for the benefit of establishing a pattern of behavior that is most accurate and consistent to the actual object’s behavior. For instance, if an action only occurs once after a condition while another action occurs one hundred times after a condition, then it would be preferred to choose the latter action as the expected action.
Regarding claim 10, Kolen in view of Ecovacs, Yamauchi, Seo, and Gomez teaches the server of claim 8. Kolen further teaches wherein the at least one processor is further configured to execute the instructions (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”) to: transmit, to the metaverse server, information for the action tree (Paragraph 0032, 0057, 0070, 0091 – “the portion of the game application 104 executed by application host systems 122 of the interactive computing system 120 may create a persistent virtual world. This persistent virtual world or virtual environment may enable one or more players to interact with the virtual world and with each other in a synchronous and/or asynchronous manner…the game may be a massively multiplayer online role-playing game (MMORPG) that includes a client portion executed by the player computing system 102 and a server portion executed by one or more application host systems 122…the custom character system 130 may generate one or more custom behavior models…the custom character system 130 may provide the virtual character data and corresponding clothing item data to the application host systems 122 and/or game applications 104 for incorporation into a virtual world”; Note: the application host system is a server, and the virtual character data, which includes the custom behavior model, is transmitted to the application host system. The virtual world is equivalent to the metaverse, and it is created by the application host system. Thus, the application host system is a metaverse server. The custom behavior model is information for the action tree), and transmit, to the electronic device, the information for the action tree ((Paragraph 0100 – “the behavior learning system 134 may use the custom behavior model (optionally combined with generic presets) when animating the virtual character in a game, such as game application 104”; Note: the custom behavior model, which is information for the action tree, is transmitted to the game application, which is part of the player’ s computing system; see screenshot of Fig. 1 below).
PNG
media_image3.png
397
560
media_image3.png
Greyscale
Screenshot of Fig. 1 (taken from Kolen)
Regarding claim 18, Kolen in view of Ecovacs and Yamauchi teaches the method of claim 11. Kolen further teaches obtaining object action information for the object (Paragraph 0094 – “the behavior learning system 134 may extract or determine traits and typical action sequences of the person depicted in the input media based on an automated analysis of the input media”); generating an observed action pattern of the object indicating that a consistent action of the object is detected by performing a rule analysis based on the object action information (Paragraph 0095 – “the behavior learning system 134 may analyze a long video file or stream of video data (such as many minutes or many hours), label time stamps or frames in the video in which particular behaviors or actions are identified, link successive behaviors or actions as action sequences (e.g., sleep, then stand up, yawn, then walk with a limp briefly before walking more easily, etc.), then compile statistics of the frequency and length of the various observations in order to apply a Markov analysis or model to predict behavior of the person”); and generating an action tree of the object based on the observed action pattern (Paragraph 0096 – “the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media, which may be in the form of behavior trees, action sequences, and/or other formats”). Kolen does not teach obtaining object action information for the object from the robot cleaner; obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server; nor “a data pattern of the IoT information” from the limitation: “generating an observed action pattern of the object indicating that a consistent action of the object is detected corresponding to a data pattern of the IoT information, by performing a rule analysis based on the object action information and the IoT information”. However, Seo teaches obtaining object action information for the object from the robot cleaner (Paragraph 0060 – “the cleaning robot 100 can grasp the position or movement of an animal based on the received direction when the sound of the animal is received through the microphone 265”; Note: the robot cleaner can collect information on movement of an animal, which is equivalent to object action information). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Seo to receive action information from the robot cleaner because robot cleaners have become common for households, making it easy for users to capture data of themselves and objects around the house. Additionally, many robot cleaners already have sensors for detecting objects and motion, so it would be convenient to receive action information from a robot cleaner. Kolen modified by Seo still does not teach obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server; nor “a data pattern of the IoT information” from the limitation: “generating an observed action pattern of the object indicating that a consistent action of the object is detected corresponding to a data pattern of the IoT information, by performing a rule analysis based on the object action information and the IoT information”. However, Gomez teaches obtaining internet of things (IoT) information from at least one electronic device among electronic devices managed by the server (Paragraph 0017, 0035-0037 – “The devices used can be Internet of Things (IoT) devices…The method also includes the following steps performed by the server: Collect information received from one or more of the monitoring devices of the group of living beings; Collect information received from one or more external data sources”); and generating an observed action pattern of the object indicating that a consistent action of the object is detected corresponding to a data pattern of the IoT information based on the object action information and the IoT information (Paragraph 0038 – “Generate behavioral patterns for each monitored device (i.e., monitoring device) based on statistical processing of the collected information”; Note: the monitored device is an IoT device. Statistical processing allows for detection of consistent action). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to obtain IoT information and use it to generate an action pattern because being connected to the Internet makes it easy for IoT devices to track data that changes overtime, which can include behaviors. This makes IoT devices a useful tool in monitoring users, and the data it collects can be utilized for other purposes, like managing health or even in a metaverse.
Regarding claim 19, Kolen in view of Ecovacs, Yamauchi, Seo, and Gomez teaches the method of claim 18. Kolen further teaches wherein the generating the observed action pattern comprises: generating condition information (Paragraph 00096-0097 – “the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time. The behavior learning system 134 may incorporate these statistics or likelihoods of certain actions into a stored behavior model…individual behaviors or actions in the behavior model may be linked or associated with environmental stimuli or objects. For example, the behavior data may indicate that the particular person is very likely to give other people a high five when seeing them for the first time in a video”; Note: in the examples given, standing up from a seated position and seeing someone for the first time are condition information), identifying a specific action based on the object action information (Paragraph 0096 – “the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media…For example, the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time”; Note: in the example, a specific action (stretching) is identified based on the object action information), and generating the observed action pattern by associating the condition information and a specific action (Paragraph 0096 – “the behavior learning system 134 may generate one or more custom character behavior models based at least in part on differences between the generic behavior model(s) obtained at block 502 and behavior data extracted or determined from the input media analysis at block 504. For example, the behavior learning system 134 either as part of block 506 or block 508, may generate a list of actions, behaviors, sequences of actions, or similar behavior information observed of the real person in the input media, which may be in the form of behavior trees, action sequences, and/or other formats. For example, the behavior learning system 134 may have determined that immediately after standing from a seated position, a particular person stretches 60% of the time and adjusts their clothing 40% of the time. The behavior learning system 134 may incorporate these statistics or likelihoods of certain actions into a stored behavior model, such as a behavior tree”; Note: the custom character behavior model is equivalent to the observed action pattern. Conditions and actions are associated. For example, the condition of standing up is associated with the action of stretching). Kolen does not teach generating condition information based on time information and the IoT information nor identifying a specific action having a ratio greater than or equal to a threshold value in the condition information, based on the object action information. However, Gomez teaches generating condition information based on time information and the IoT information (Paragraph 0030-0031 – “a) Receive information on one or more physical, physiological and/or environmental variables of the living being carrying the monitoring device, measured by one or more sensors associated with the monitoring device; b) Obtain statistical parameters for each variable in time windows by processing the information received;”; Note: the time windows and the associated monitoring device are the condition information) and identifying a specific action having a ratio greater than or equal to a threshold value in the condition information, based on the object action information (Paragraph 0030-0033 – “a) Receive information on one or more physical, physiological and/or environmental variables of the living being carrying the monitoring device, measured by one or more sensors associated with the monitoring device; b) Obtain statistical parameters for each variable in time windows by processing the information received; c) Compare the value of each statistical parameter in the current time window with thresholds previously calculated by the monitoring device, based at least on previous values of said statistical parameters; d) If as a result of such comparison, no anomaly is detected in any of the statistical parameters, recalculate the value of each of the thresholds taking into account the current value of each statistical parameter in the current time window”; Note: the variable of the living being is equivalent to the action, and the specific variable can be identified to have a statistical value and is compared with other previous values, as a threshold, to evaluate its normality). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to generate conditions based on IoT and time information because observing IoT usage and time allows the action pattern to reflect the object/user’s schedule and track their common behaviors, which would then allow for an accurate representation of the object/user. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Gomez to identify a specific action exceeding a threshold of the condition for the benefit of establishing a pattern of behavior that is most accurate and consistent to the actual object’s behavior. For instance, if an action only occurs once after a condition while another action occurs one hundred times after a condition, then it would be preferred to choose the latter action as the expected action.
Regarding claim 27, Kolen in view of Ecovacs, Yamauchi, Seo, and Gomez teaches the server of claim 8. Kolen further teaches wherein the at least one processor is further configured to execute the instructions to (Paragraph 0122 – “All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors”): identify a default action pattern of condition information, the default action pattern corresponding to a preconfigured action pattern of the object (Paragraph 0093, 0096 – “The method 500 begins at block 502, where the custom character system 130 obtains input media depicting a real person, as well as one or more generic behavior model(s). Obtaining input media has been described in detail above. The behavior model data and/or related animation data obtained may be in a variety of formats, such as behavior trees, rule sets, skeletal movement data, lists of action sequences, kinematic chains and/or other kinematic data, and/or other animation-related or behavior-related data… a generic or base behavior model for a generic human character may already include hundreds of potential actions or behaviors arranged in a behavior tree”; Note: the generic behavior model contains a default action pattern), and based on determining that the default action pattern of the condition information is different from the observed action pattern associated with the condition information, delete the default action pattern in the action tree (Paragraph 0096, 0101 – “the custom behavior model may be generated as a modified version of the base behavior model. For example, a generic or base behavior model for a generic human character may already include hundreds of potential actions or behaviors arranged in a behavior tree, and the behavior learning system 134 may alter the propensities for the customized virtual character to engage in particular actions in particular orders when there is significant deviation from the standard behaviors and the particular observed person's behaviors… the behavior learning system 134 may optionally adjust the custom behavior model based on in-game user feedback in order to enforce correct behavior and suppress incorrect behavior within the model”; Note: it is implied that a default action pattern is deleted from the action tree when there is deviation from the observed action pattern since the generic behavior model is altered to contain only correct behaviors. The behavior model is an action tree).
Claims 22 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Kolen in view of Ecovacs, Yamauchi, and Baumberg (US 7006089 B2), hereinafter Baumberg.
Regarding claim 22, Kolen in view of Ecovacs and Yamauchi teaches the server of claim 1. Kolen does not teach wherein the rendering information is generated based on at least one of the first image information with a first weight or the second image information with a second weight, wherein, in a case that the third area is identified from the at least one first image, the first weight is greater than the second weight, and wherein, in a case that the third area is not identified from the at least one first image, the first weight is lower than or equal to the second weight. However, Baumberg teaches wherein the rendering information is generated based on at least one of the first image information with a first weight or the second image information with a second weight (Col. 3 lines 44-49 – “As part of generation of the texture maps, the surface texturer 36 initially determines weight function data indicative of the relative reliability of different input images for generating texture maps for texture rendering different portions of a modelled object”; Note: rendering information is generated based on the weight of images), wherein, in a case that the third area is identified from the at least one first image, the first weight is greater than the second weight (Col. 15 lines 30-38 – “greater weight is placed on images in which portions of a model are most easily viewed giving the resultant canonical low frequency images a realistic appearance and a neutral tone”; Note: when portions of an object/model are identified from an image, that image has greater weight than other images), and wherein, in a case that the third area is not identified from the at least one first image, the first weight is lower than or equal to the second weight (Col. 15 lines 30-38 – “greater weight is placed on images in which portions of a model are most easily viewed giving the resultant canonical low frequency images a realistic appearance and a neutral tone”; Note: when portions of an object/model are not identified from an image, that image has less weight than other images). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Baumberg to render based on weights of the images and give greater weight when a portion of the object is identified in the image for the benefit of “giving the resultant canonical low frequency images a realistic appearance and a neutral tone” (Baumberg: Col. 15 lines 30-38). In other words, a realistic and visually appealing rendering is achieved as a result of putting more emphasis on the most relevant images. This would benefit the avatar generation in Kolen since it would help create an avatar that best resembles the user.
Regarding claim 25, Kolen in view of Ecovacs and Yamauchi teaches the method of claim 11. Kolen does not teach wherein the rendering information is generated based on at least one of the first image information with a first weight or the second image information with a second weight, wherein, in a case that the third area is identified from the at least one first image, the first weight is greater than the second weight, and wherein, in a case that the third area is not identified from the at least one first image, the first weight is lower than or equal to the second weight. However, Baumberg teaches wherein the rendering information is generated based on at least one of the first image information with a first weight or the second image information with a second weight (Col. 3 lines 44-49 – “As part of generation of the texture maps, the surface texturer 36 initially determines weight function data indicative of the relative reliability of different input images for generating texture maps for texture rendering different portions of a modelled object”; Note: rendering information is generated based on the weight of images), wherein, in a case that the third area is identified from the at least one first image, the first weight is greater than the second weight (Col. 15 lines 30-38 – “greater weight is placed on images in which portions of a model are most easily viewed giving the resultant canonical low frequency images a realistic appearance and a neutral tone”; Note: when portions of an object/model are identified from an image, that image has greater weight than other images), and wherein, in a case that the third area is not identified from the at least one first image, the first weight is lower than or equal to the second weight (Col. 15 lines 30-38 – “greater weight is placed on images in which portions of a model are most easily viewed giving the resultant canonical low frequency images a realistic appearance and a neutral tone”; Note: when portions of an object/model are not identified from an image, that image has less weight than other images). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kolen to incorporate the teachings of Baumberg to render based on weights of the images and give greater weight when a portion of the object is identified in the image for the benefit of “giving the resultant canonical low frequency images a realistic appearance and a neutral tone” (Baumberg: Col. 15 lines 30-38). In other words, a realistic and visually appealing rendering is achieved as a result of putting more emphasis on the most relevant images. This would benefit the avatar generation in Kolen since it would help create an avatar that best resembles the user.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ebrahimi et al. (US 20210089040 A1) teaches a robot cleaner that captures images of the workspace, generates a representation of the workspace, and has an avatar that represents itself.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617