DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 17 March 2026 has been entered.
Response to Arguments
Applicant’s arguments have been fully considered but they are moot in view of the new grounds of rejection presented in this Office Action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 6, 7, 9, 11, 12, 17, 18, 22, 23, 25, 27, 28, 31, and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Dsouza et al. (US 2019/0355172; hereinafter “Dsouza”) in view of LeGendre et al. (US 2024/0212106; hereinafter “LeGendre”).
Regarding claim 1, Dsouza discloses A method comprising: determining an estimated camera pose corresponding to a frame of image data (“the orientation, position, and/or movement of the image capturing device (e.g., mobile devices),” para. 27); generating a background replacement view of a configured three-dimensional (3D) content (“to generate the background image, both 3D and 2D image resources may be used,” para. 36; “The background image may be generated from the selected image resource based at least in part on the determined orientation of the mobile device … the image resource is a three-dimensional (3D) world model for background, e.g., obtained from 3D modeling or 3D reconstruction,” paras. 54-55), wherein the background replacement view is associated with an angle-of-view (AOV) based on the estimated camera pose (“the virtual backgrounds can be generated based on the orientation, position, and/or location of the mobile device,” para. 45); determining a segmentation mask for the frame of image data, the segmentation mask indicative of a foreground portion of the frame of image data and a background portion of the frame of image data (“a foreground image is segmented from the original image … a transparency mask (e.g., a binary mask) for the foreground object is produced,” para. 57); and generating an output image frame based on the … image and the background replacement view of the configured 3D content (“obtain … a foreground image, a foreground mask, and a virtual background image and use such components to generate a composite image,” para. 38).
Dsouza does not disclose generating a relighting image corresponding to at least a portion of the frame of image data, wherein the relighting image is generated based on applying a light map to the foreground portion of the frame of image data, and wherein the light map is based on lighting information of the configured 3D content and information indicative of learned surface characteristics of the foreground portion or generating the output image frame based on the relighting image.
In the same art of compositing a foreground image with a background scene, LeGendre teaches generating a relighting image corresponding to at least a portion of the frame of image data, wherein the relighting image is generated based on applying a light map to the foreground portion of the frame of image data, and wherein the light map is based on lighting information of the configured 3D content and information indicative of learned surface characteristics of the foreground portion (“Relighting network 200 [of Fig. 2] calculates a per-pixel lighting representation or light maps L, which encode the specular S and diffuse D components of surface reflection for a given omnidirectional target HDR lighting environment and inferred surface geometry … predicting, via the neural network, an output image comprising the subject in the relit foreground … neural shading net 260 performs the final foreground rendering to generate relit foreground,” para. 49) and generating the output image frame based on the relighting image (“Compositing component 755 [of Fig. 7] generates a composite of relit foreground 745 into a new background,” para. 72).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of LeGendre to Dsouza. The motivation would have been “users may prefer to have access to realistic, professional, and high-quality image relighting and compositing features” (LeGendre, para. 4).
Regarding claim 2, the combination of Dsouza and LeGendre renders obvious wherein: a foreground portion of the output image frame comprises pixel data included in the relighting image (“predicting, via the neural network, an output image comprising the subject in the relit foreground,” LeGendre, para. 49; see claim 1 for motivation to combine); and a background portion of the output image frame comprises pixel data included in the background replacement view of the configured 3D content (“obtain … a foreground image, a foreground mask, and a virtual background image and use such components to generate a composite image,” Dsouza, para. 38).
Regarding claim 6, the combination of Dsouza and LeGendre renders obvious wherein the frame of image data is a frame of video data included in a plurality of frames of video data (“the existing or original background images, for example in a video, can be replaced with a virtual background associated with an orientation or position of the mobile device,” Dsouza, para. 19).
Regarding claim 7, the combination of Dsouza and LeGendre renders obvious wherein the lighting information (e.g. “a target HDR illumination” of LeGendre, para. 53; see claim 1 for motivation to combine) is included in an equirectangular projection of the configured 3D content (“rasterizes a part of an equirectangular projection of the panoramic image to form the virtual background image,” Dsouza, para. 22).
Regarding claim 9, the combination of Dsouza and LeGendre renders obvious wherein the configured 3D content is a computer-generated (CG) 3D model of a scene selected from a plurality of CG 3D models of scenes (“The user may browser various image resources … the image resource is a three-dimensional (3D) world model for background, e.g., obtained from 3D modeling or 3D reconstruction,” Dsouza, paras. 54-55).
Regarding claim 11, the combination of Dsouza and LeGendre renders obvious wherein the background replacement view of the CG 3D scene is generated using an image capture device used to obtain the frame of image data (“system 200 [of Fig. 2] is embodied as a specialized computing device in a mobile phone,” Dsouza, para. 39; “the mobile device has a camera,” Dsouza, para. 50; Fig. 2 of Dsouza illustrates the Background Processor 230 being included within the mobile device).
Regarding claim 12, the combination of Dsouza and LeGendre renders obvious wherein: the background replacement view comprises a view of a portion of the CG 3D scene, wherein the view of the portion of the CG 3D scene is associated with an AOV corresponding to the frame of image data and the estimated camera pose (“The background image may be generated from the selected image resource based at least in part on the determined orientation of the mobile device … the image resource is a three-dimensional (3D) world model for background … rasterize a part of the panoramic image to generate the background image,” Dsouza, paras. 54-56).
Regarding claim 17, it is rejected using the same citations and rationales described in the rejection of claim 1, with the additional limitations of An apparatus for processing image data, comprising: at least one memory; and at least one processor coupled to the at least one memory (see Dsouza, Fig. 8).
Regarding claims 18, 22, 23, 25, 27, and 28, they are rejected using the same citations and rationales described in the rejections of claims 2, 6, 7, 9, 11, and 12, respectively.
Regarding claim 31, the combination of Dsouza and LeGendre renders obvious generating the light map based on the lighting information of the configured 3D content and the learned surface characteristics of the foreground portion of the frame of image data, wherein the light map is indicative of lighting influence between the configured 3D content and a surface associated with the foreground portion of the frame of image data (“maintaining a consistent lighting of the foreground with a target illumination of target background,” LeGendre, para. 72; see claim 1 for motivation to combine; this shows that the foreground light maps are influenced by the background lighting, and the background of Dsouza is configured 3D content; therefore the combination of references renders obvious the configured 3D content background of Dsouza influencing the foreground light maps in order to maintain consistent lighting).
Regarding claim 32, it is rejected using the same citations and rationales described in the rejection of claim 31.
Claims 3-5 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Dsouza and LeGendre, and further in view of Liang et al. (US 2018/0115714; hereinafter “Liang”).
Regarding claim 3, the combination of Dsouza and LeGendre does not disclose performing camera motion stabilization to generate: a stabilized camera pose, the stabilized camera pose based on one or more of the estimated camera pose or inertial sensor data corresponding to an image capture device used to obtain the image data; and a stabilization warp grid corresponding to one or more stabilization corrections determined for the image data.
In the same art of using handheld mobile devices for recording images and video, Liang teaches performing camera motion stabilization to generate: a stabilized camera pose (“continuously model a virtual (i.e., corrected or stabilized) camera orientation,” para. 20), the stabilized camera pose based on one or more of the estimated camera pose or inertial sensor data corresponding to an image capture device used to obtain the frame of image data (“video stabilization may be achieved by correlating gyroscope data and focal length sensor data of an image capture device,” para 20); and a stabilization warp grid corresponding to one or more stabilization corrections determined for the frame of image data (“a grid or stabilization mesh can be used to transform the pixels of an input frame associated with the real camera orientation to an output frame associated with the virtual camera orientation, such that the video may be stabilized,” para. 20).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Liang to the combination of Dsouza and LeGendre. The motivation would have been to “more accurately model the motion of the camera to remove undesired movements” (Liang, para. 4).
Regarding claim 4, the combination of Dsouza, LeGendre, and Liang renders obvious wherein the background replacement view comprises a view of the configured 3D content rendered using an AOV corresponding to the stabilized camera pose (para. 45 of Dsouza teaches generating a background replacement view based on camera pose, and para. 20 of Liang renders obvious using a stabilized camera pose; see claim 3 for motivation to combine).
Regarding claim 5, the combination of Dsouza, LeGendre, and Liang renders obvious applying the one or more stabilization corrections to the frame of image data, based on combining the stabilization warp grid and the frame of image data to generate a stabilized frame of image data (“a grid or stabilization mesh can be used to transform the pixels of an input frame,” Liang, para. 20; see claim 3 for motivation to combine); and generating the relighting image based on combining at least a portion of the stabilized frame of image data and the lighting information (para. 49 of LeGendre teaches generating the relighting image based on image data and lighting information, and para. 20 of Liang renders obvious using stabilized image data; see claims 1 and 3 for motivation to combine).
Regarding claims 19-21, they are rejected using the same citations and rationales described in the rejections of claims 3-5, respectively.
Claims 8, 13-15, 24, 29, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Dsouza and LeGendre, and further in view of Wagner et al. (US 2015/0294499; hereinafter “Wagner”).
Regarding claim 8, the combination of Dsouza and LeGendre does not specifically recite wherein the estimated camera pose comprises an estimated 6 degrees-of-freedom (6DOF) pose of a camera used to capture the frame of image data, at a time of capture of the frame of image data.
In the same art of image capturing and 3D modeling, Wagner teaches wherein the estimated camera pose comprises an estimated 6 degrees-of-freedom (6DOF) pose of a camera used to capture the image data, at a time of capture of the image data (“camera pose may be determined for 6-Degrees Of Freedom (6DOF),” para. 39).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Wagner to Dsouza and LeGendre. The motivation would have been “for power efficient 3D reconstruction” (Wagner, para. 47).
Regarding claim 13, the combination of Dsouza and LeGendre does not disclose wherein the configured 3D content is based on an additional frame of image data captured using a same image capture device used to obtain the image data.
In the same art of mobile device image capture and 3D modeling, Wagner teaches wherein the configured 3D content is based on an additional frame of image data captured using a same image capture device used to obtain the image data (“MS [Mobile Station] 100 [of Fig. 1] may comprise multiple cameras 110, such as dual front cameras and/or a front and rear-facing camera,” para. 36; “processing one or more images captured by camera 110 to perform 3D reconstruction of an environment being modeled,” para. 45).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Wagner to the combination of Dsouza and LeGendre. Dsouza teaches using 3D models for generating a background viewpoint, but does not disclose that the 3D models were generated on the mobile device from images captured by the mobile device. Wagner remedies this deficiency by teaching that 3D models can be generated directly on a mobile device from images captured by the mobile device. The motivation would have been “to facilitate power efficient real-time 3D reconstruction on computing and mobile devices” (Wagner, para. 6).
Regarding claim 14, the combination of Dsouza, LeGendre, and Wagner renders obvious wherein the frame of image data and the additional frame of image data are captured using a same camera of the image capture device (“MS [Mobile Station] 100 [of Fig. 1] may comprise multiple cameras 110, such as dual front cameras and/or a front and rear-facing camera,” Wagner, para. 36; “processing one or more images captured by camera 110 to perform 3D reconstruction of an environment being modeled,” Wagner, para. 45; see claim 13 for motivation to combine; Wagner renders obvious capturing images from any or all of the cameras on a mobile device, including a same camera).
Regarding claim 15, the combination of Dsouza, LeGendre, and Wagner renders obvious wherein the frame of image data is captured using a first camera of the image capture device, and wherein the additional frame of image data is captured using a second camera of the image capture device (“MS [Mobile Station] 100 [of Fig. 1] may comprise multiple cameras 110, such as dual front cameras and/or a front and rear-facing camera,” Wagner, para. 36; “processing one or more images captured by camera 110 to perform 3D reconstruction of an environment being modeled,” Wagner, para. 45; see claim 13 for motivation to combine; Wagner renders obvious capturing images from any or all of the cameras on a mobile device, including using different cameras).
Regarding claims 24, 29, and 30, they are rejected using the same citations and rationales described in the rejections of claims 8, 13, and 14, respectively.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Dsouza, LeGendre, and Wagner, and further in view of Tsuchiya (US 2025/0139879).
Regarding claim 16, the combination of Dsouza, LeGendre, and Wagner renders obvious wherein: the background replacement view comprises a viewport in the additional frame of image data, wherein an AOV of the viewport is a subset of an AOV of the additional frame of image data (“The background image may be generated from the selected image resource based at least in part on the determined orientation of the mobile device … rasterize a part of the panoramic image to generate the background image,” Dsouza, paras. 54-56).
The combination of Dsouza, LeGendre, and Wagner does not specifically recite the AOV of the viewport is determined based on camera intrinsic information associated with the image capture device.
In the same art of combining a foreground image with background 3D content, Tsuchiya teaches the AOV of the viewport is determined based on camera intrinsic information associated with the image capture device (“the viewpoint position with respect to the 3D background data is specified on the basis of the position, the imaging direction, the angle of view, or the like of the camera to be reflected in the current frame, and rendering is performed. At this time, video processing reflecting the focal length, the F-number, the shutter speed, the lens information, or the like can also be performed,” para. 120).
Before the effective filing date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Tsuchiya to the combination of Dsouza, LeGendre, and Wagner. The motivation would have been “to improve the overall video production efficiency” (Tsuchiya, para. 362).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan McCulley whose telephone number is (571) 270-3754. The examiner can normally be reached Monday through Friday, 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN MCCULLEY/Primary Examiner, Art Unit 2611
Prosecution Insights