DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments filed on 08 June 25, 2026 have been entered. Claims 1, 2, 5-14, and 16-19 have been amended. Claims 4, 15, 20-22 have been canceled. Claims 23-25 have been added. Claims 1-3, 5-14, 16-19 and 23-25 are still pending in this application, with claims 1, 11 and 16 being independent.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05 November, 2025 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 6, 7, 9, 11, 13, 16-19, 23 and 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over IDS Snap Inc. (US 20210065454 A1), referred herein as HAN et al. (US 20190392632 A1), referred herein as HAN in view of Kopeinigg et al. (US 20200312037 A1), referred herein as Kopeinigg.
Regarding claim 1, HAN in view of Kopeinigg teaches an apparatus comprising:
at least one processor configured to (HAN FIG. 1.120: processor module):
extract 2D image information from a 2D image showing at least one object that has at least a first feature and a background that has at least a second feature (HAN [0072] the background image 10 may include a plurality of depth images or a plurality of color images captured at different viewpoints; [0078] The object extraction module 120_1 may be a module for performing the object extraction process and may perform a process of extracting a foreground including an object from a first object image 12 obtained through coarse object scanning by using a 3D background model. [0118] Referring to FIG. 4, first, in step S410, the processor module 120 or the object extraction module 120_1 of the processor module 120 may perform a process of extracting a foreground image including the object from the first object image (or a depth image) obtained through the first object scanning process (or the coarse scanning process). For example, the foreground image may be extracted by removing a background image; FIG. 4.S420: extract first feature data; [0134] a process of extracting 2-2.sup.th feature data from a 2D object image where the 3D object model reconstructed based on the first object image is projected onto a 2D plane may be performed by using the feature extraction algorithm); background depth and/or color image is interpreted as the second feature.
match the first feature to at least a first 3D model and the second feature to at least a second 3D model (HAN FIG. 4.S440: reconstruct 3D object model data based on first pose data; FIG. 3.320: reconstruct 3D background model);
HAN in view of Kopeinigg teaches
customize the first 3D model based on the first feature (Kopeinigg [0048] Custom mesh 204 may be formed around skeletal model 602 based on the application of various parameters 216 to parametric 3D model 220) and the second 3D model based on the second feature (HAN [0125] the learning data stored in the storage module 130 may be continuously updated according to control by the processor module 120, and whenever the learning data is updated, learning data obtained through the updating may be reflected in real time in reconstructing the final 3D object model obtained through the second object scanning process);
generate an animation that combines the customized first 3D model and the customized second 3D model based on the 2D image information (HAN [0115] Subsequently, in step S360, the final 3D object model may be reconstructed based on the second object image obtained through the second object scanning process and the learning data which is generated through the pose learning in step S350.); and
cause a presentation of the animation (Kopeinigg [0061] Referring back to FIG. 2, dataflow 200 shows that, once created, custom 3D model 208 may be processed at animation stage 210 together with motion capture video 222 to form presentation data 224).
Kopeinigg discloses a volumetric capture system that accesses a two-dimensional (“2D”) image captured by a capture device and depicting a first subject of a particular subject type, which is analogous to the present patent application.
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the customed three-dimensional (“3D”) animation model, as suggested by Kopeinigg into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so, various useful animation, entertainment, educational, vocational, communication, and/or other applications may be implemented and deployed.
Regarding claim 2, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the processor is further configured to:
receive from at least one input device at least one interaction signal (HAN [0065] The user interface module 150 may include one or more buttons, microphones, and speakers. The user interface module 150 may generate a scanning start command and a scanning end command, based on a user input. The user interface module 150 may be used for interacting with the textured mesh of the object); and
alter presentation of the 3D interactive video at least in part based on the interaction signal (HAN [0125] the learning data stored in the storage module 130 may be continuously updated according to control by the processor module 120, and whenever the learning data is updated, learning data obtained through the updating may be reflected in real time in reconstructing the final 3D object model obtained through the second object scanning process).
Regarding claim 3, HAN in view of Kopeinigg teaches the apparatus of Claim 2, and further teaches wherein the at least one input device comprises at least one touch element on at least one computer simulation controller (HAN [0064] The reconstructed 3D model may be displayed on a display screen included in the display module 140. The user may touch the display screen to check a 3D model of an object while laterally, upward, and downward rotating the 3D model displayed on the display screen; FIG. 2.120_9: pose learning module).
Regarding claim 6, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the at least one first feature of the 2D image information comprises human image texture (Kopeinigg [0025] FIG. 3 shows illustrative aspects of how 2D image data 212 is captured by an illustrative capture device. As shown in FIG. 3, 2D imagery associated with a first subject 302 may be captured by a capture device 304 by aiming a field of view 306 of capture device 304 in the direction of the subject. In this example, the first subject 302 is shown to be a human subject, a young girl in this example).
Regarding claim 7, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the at least one first feature of the 2D image information comprises human image motion (Kopeinigg [0023] processing facility 104 may animate, based on the motion capture video; [0030] a machine learning system 402 that generates machine learning model 214 (the machine learning model illustrated in FIG. 2 above) based on input training data 404 and training that may involve human input).
Regarding claim 9, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the at least one first feature of the 2D image information comprises environment type (Kopeinigg [0002] by creating a model of a subject (e.g., a person, an animal, an inanimate object, etc.) that is present in a real-world environment, a system may provide an augmented reality experience involving the subject to the user; [0015] Such behaviors may themselves be captured using a single capture device when the behaviors are performed by the same or a different subject of the same subject type (e.g., a professional dancer, an actor, etc.)).
Regarding claim 11, HAN in view of Kopeinigg teaches a non-transitory computer-readable medium storing instructions that, when executed by of a computing device, cause the one or more processors computing device to [perform] (HAN FIG. 1.120: processor module; Kopeinigg [0076] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer)).
The metes and bounds of the claim substantially correspond to the limitations as set forth in Claim 1; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding claim 13, HAN in view of Kopeinigg discloses the non-transitory computer-readable medium of Claim 11. The metes and bounds of the claim substantially correspond to the limitations as set forth in Claim 2; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding claim 16, HAN in view of Kopeinigg teaches a method, comprising:
The metes and bounds of the claim substantially correspond to the limitations as set forth in Claim 1; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding claim 17, HAN in view of Kopeinigg teaches the method of Claim 16, and further teaches further comprising: animating at least one character in the 3D video based on action in the 2D images (Kopeinigg [0004] FIG. 1 shows an illustrative volumetric capture system for generating an animated three-dimensional (“3D”) model based on a two-dimensional (“2D”) image).
Regarding claim 18, HAN in view of Kopeinigg teaches the method of Claim 16, and further teaches further comprising: playing audible dialog in the 3D video based at least in part on an image of a user viewing the 3D video (Kopeinigg [0064] motion capture video 222 may be selected from a library of motion capture videos. For example, such a library could include a variety of different dances set to different songs, a variety of action stunts performed using different props or scenery, or the like; HAN [0072] First, in order to reconstruct a 3D background model (or 3D background model data), a background image 10 obtained based on background scanning may be input to the pose estimation module 120_7. Here, the background image 10 may include a plurality of depth images or a plurality of color images captured at different viewpoints).
Regarding claim 19, HAN in view of Kopeinigg teaches the method of Claim 16, and further teaches further comprising: animating at least one character in the 3D video based on input from an input device (HAN [0065] The user interface module 150 may include one or more buttons, microphones, and speakers. The user interface module 150 may generate a scanning start command and a scanning end command, based on a user input. The user interface module 150 may be used for interacting with the textured mesh of the object).
Regarding claim 23, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the first feature comprises a human texture and the at least a first 3D model comprises a 3D character model, and wherein the second feature comprises an environmental background type and the at least a second 3D model comprises a 3D environmental background model (Kopeinigg [0025] FIG. 3 shows illustrative aspects of how 2D image data 212 is captured by an illustrative capture device. As shown in FIG. 3, 2D imagery associated with a first subject 302 may be captured by a capture device 304 by aiming a field of view 306 of capture device 304 in the direction of the subject. In this example, the first subject 302 is shown to be a human subject, a young girl in this example; HAN [0109] Referring to FIG. 3, first, in step S310, background scanning may be performed by the camera module 110. A background image 10 may be obtained through the background scanning In this case, the background image 10 may include a plurality of images having different viewpoints, and each of the plurality of images may be a depth image or a color image; [0110] Subsequently, in step S320, the processor module 120 may perform a process of reconstructing a 3D background model (or 3D background model data), based on the background image 10).
Regarding claim 25, HAN in view of Kopeinigg teaches the apparatus of Claim 1, and further teaches wherein the at least one processor is further configured to: update the customized first 3D model over a plurality of time periods as additional 2D images are added to a memory (HAN [0041] An object image obtained based on the first object scanning process may be used as an image for obtaining 3D model data and pose data used for initial learning. The object image obtained based on the first object scanning process may include a plurality of color images or a plurality of depth images having different viewpoints; Kopeinigg [0062] FIG. 8 shows illustrative aspects of how custom 3D model 208 may be animated at animation stage 210 of system 100. As has been described, custom 3D model 208 may be generated from 2D image data 212 (e.g., a single image, a plurality of images, a video image, etc.) where subject 302 poses for the model creation; [0065] a library of motion capture videos 222, respective parametric model animations 802 associated with each motion capture video 222 may be stored in connection with the library in a database), wherein generating the animation is based on a selected time period of the plurality of time periods (Kopeinigg [0064] motion capture video 222 may be selected from a library of motion capture videos. For example, such a library could include a variety of different dances set to different songs, a variety of action stunts performed using different props or scenery, or the like). Motion capture video integrates time and motion.
Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over IDS Snap Inc. (US 20210065454 A1), referred herein as HAN et al. (US 20190392632 A1), referred herein as HAN in view of Kopeinigg et al. (US 20200312037 A1), referred herein as Kopeinigg and ERONEN et al. (US 20180276476 A1), referred herein as ERONEN.
Regarding claim 5, HAN in view of Kopeinigg discloses the apparatus of Claim 1. However, ERONEN teaches wherein the processor is further configured to: alter presentation of the 3D interactive video responsive to an incoming telephone call (ERONEN [0080] the event 201 may be a notification that alerts a user to a system-state. For example, a notification may be a system notification relating to a personal device, system or service of the user such as a programmed alarm, a calendar alert etc. For example, a notification may be a communication notification, for example, indicating there is an incoming telephone call or an incoming electronic message or some other communication event).
ERONEN discloses rendering of media via virtual reality or augmented reality, which is analogous to the present patent application.
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the media rendering process through the communication of telephone calls, as suggested by ERONEN into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so, the system can alert the user of emergency notifications during an interactive media session.
Regarding claim 14, HAN in view of Kopeinigg discloses the non-transitory computer-readable medium of Claim 11. The metes and bounds of the claim substantially correspond to the limitations as set forth in Claim 5; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Claim(s) 8, 10, 12 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over IDS Snap Inc. (US 20210065454 A1), referred herein as HAN et al. (US 20190392632 A1), referred herein as HAN in view of Kopeinigg et al. (US 20200312037 A1), referred herein as Kopeinigg and Goodrich et al. (US 20210065454 A1), referred herein as Goodrich.
Regarding claim 8, HAN in view of Kopeinigg discloses the apparatus of Claim 1. However, Goodrich teaches wherein the at least one first feature of the 2D image information comprises human image facial emotion (Goodrich [0080] he complex image manipulations may include size and shape changes, emotion transfers (e.g., changing a face from a frown to a smile)).
Goodrich discloses a technology that applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, which is analogous to the present patent application.
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the emotion transfer data, as suggested by Goodrich into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so would enhance users' experiences with digital images and provide various features, and enable computing devices to perform image processing operations on various objects and/or features captured in a wide range of changing conditions.
Regarding claim 10, HAN in view of Kopeinigg discloses the apparatus of Claim 1. However, Goodrich teaches wherein the processor is further configured to: alter presentation of the 3D interactive video responsive to input from a time slider input element (Goodrich [0148] At operation 812, the sharing module 712 stores at or sends the generated 3D message to the messaging server system 108. In an example, the messaging client application 104 sends the 3D message to the messaging server system 108, which enables the 3D message to be stored and/or viewed at a later time by a particular recipient or viewer of the 3D message).
Goodrich discloses a technology that applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, which is analogous to the present patent application.
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the playback of an animation video, as suggested by Goodrich into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so would enhance users' experiences with digital images and provide various features, and enable computing devices to perform image processing operations on various objects and/or features captured in a wide range of changing conditions.
Regarding claim 12, HAN in view of Kopeinigg discloses the non-transitory computer-readable medium of Claim 11. However, Goodrich teaches wherein the instructions are executable to: present on at least one 3D display the 3D video.
Goodrich discloses a technology that applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, which is analogous to the present patent application (Goodrich [0137] The sharing module 712 generates the 3D message for storing and/or sending to the messaging server system 108. The sharing module 712 enables sharing of 3D messages to other users of the messaging server system 108).
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the 3D message, as suggested by Goodrich into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so would enhance users' experiences with digital images and provide various features, and enable computing devices to perform image processing operations on various objects and/or features captured in a wide range of changing conditions.
Regarding claim 24, HAN in view of Kopeinigg discloses the non-transitory computer-readable medium of Claim 11. However, Goodrich teaches wherein the at least one processor is further configured to: match the second feature to the at least a second 3D model by classifying the background in the 2D image using a machine learning model (Goodrich [0211] a depth inpainting mask 1500 is generated using at least the depth map. In an example, the depth inpainting mask 1500 can be determined using approaches applied on the depth map including boundary detection, and machine learning techniques such as deep convolutional neural networks that perform classifications of each pixel in the depth map, encoder-decoder architecture for segmentation). According to claim 1, the background depth map is interpreted the second feature.
It would have been obvious to one of ordinary skill in the art to have modified the systems and methods of HAN to apply the classifications of each pixel in the depth map, as suggested by Goodrich into the method of reconstructing a three-dimensional (3D) model of an object.
Doing so would enhance users' experiences with digital images and provide various features, and enable computing devices to perform image processing operations on various objects and/or features captured in a wide range of changing conditions.
Response to Arguments
Applicant’s arguments, see page 6, filed on 08 June 25, 2026, with respect to the rejection(s) of claim(s) 1, 11 and 16 under 103 rejection have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of HAN and Kopeinigg.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached on (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights