Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 12, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen (US20230231972) in view of Moon (KR20200052074).
Regarding Claim 1. Nguyen teaches An electronic device comprising:
a memory comprising a video communication application (VCA) comprising a plurality of virtual background images, (Nguyen, abstract, the invention describes system which can automatically adjust a virtual background used by a participant in a video conference to match lighting level conditions present in a foreground of a participant video stream captured at a participant device. The matching can be done by adjusting one or more virtual background parameters, including but not limited to, exposure, brightness, contrast, and ISO. In some implementations, the system can dynamically or automatically adjust the lighting levels of the virtual background based on changes that may occur to the lighting level of the foreground during the video conference.
[0011] Users of conferencing software, referred to as conference participants when participating in a conference implemented over conferencing software, enjoy using virtual backgrounds. A virtual background is a virtual image which is used in place of the actual background of the conference participant. The virtual background may be selected by the conference participant within a client-side conferencing software.
[0038] FIG. 2 is a block diagram of an example internal configuration of a computing device 200 of an electronic computing and communications system. In one configuration, the computing device 200 may implement one or more of the client 104, the application server 108, the database server 110, or the telephony server 112 of the system 100 shown in FIG. 1.
[0039] The computing device 200 includes components or units, such as a processor 202, a memory 204, a bus 206, a power source 208, peripherals 210, a user interface 212, a network interface 214, other suitable components, or a combination thereof.);
Nguyen fails to explicitly teach, however, Moon teaches each illuminated by a light source from a respective one of a plurality of different directions (Moon, abstract, the invention describes method for processing an image reflecting the location of a light source moving in augmented reality and a device thereof. To this end, one embodiment of the present invention comprises: (a) a step of which a light source unit moved by a user and a plurality of optical transceivers located at different locations transmit and receive a beam to and from each other; (b) a step of which the three-dimensional location, direction, and angle of the light source unit are checked based on the beam transmitted and received by one or both of the light source unit and the plurality of optical transceivers and three-dimensional information of the light source unit which is measured by an inertial sensor located in the light source unit; (c) a step of which a virtual object image to be displayed on an object and a background illuminated by the light source unit is processed based on the checking result of step (b); and (d) a step of which the virtual object image is displayed on one or both of the object and the background. Here, the virtual object image is characterized by being a virtual image which embodies augmented reality after being synthesized with one or both of the shadow of the object and the background generated by the object and the light source unit.
[0010] … FIGS. 3a, FIGS. 3b, FIGS. 3c, and FIGS. 3d are drawings illustrating results reflecting different light source positions in the first embodiment of the present invention illustrated in FIG. 2b.).
Nguyen and Moon are analogous art because they both teach method of rendering virtual light in a composite video including user image. Moon further teaches a plurality of light source in rendering the composite video. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the virtual light rendering method for a composite video (taught in Nguyen), to further include a plurality of light source with different directions (taught in Moon), so as to reflect the position of a moving light source in augmented reality environment (Moon, [0001]).
The combination of Nguyen and Moon further teaches a communications subsystem that links the electronic device to a communication network (Nguyen, [0039] The computing device 200 includes components or units, such as a processor 202, a memory 204, a bus 206, a power source 208, peripherals 210, a user interface 212, a network interface 214, other suitable components, or a combination thereof. One or more of the memory 204, the power source 208, the peripherals 210, the user interface 212, or the network interface 214 can communicate with the processor 202 via the bus 206.); and
a processor communicatively coupled to the memory and the communications subsystem, and which executes the VCA to enable a video communication session between the electronic device and one or more second electronic device, the processor further configured to cause the electronic device (Nguyen, [0038] FIG. 2 is a block diagram of an example internal configuration of a computing device 200 of an electronic computing and communications system. In one configuration, the computing device 200 may implement one or more of the client 104, the application server 108, the database server 110, or the telephony server 112 of the system 100 shown in FIG. 1.
[0039] The computing device 200 includes components or units, such as a processor 202, a memory 204, a bus 206, a power source 208, peripherals 210, a user interface 212, a network interface 214, other suitable components, or a combination thereof. One or more of the memory 204, the power source 208, the peripherals 210, the user interface 212, or the network interface 214 can communicate with the processor 202 via the bus 206.) to:
detect a virtual background setting of the VCA for the video communication
session:
receive a video stream from an image capturing device positioned to capture a user of the electronic device during the video communication session (Nguyen, [0015] In some implementations, a virtual background auto-correction system provides real-time adjustments of the background lighting information according to foreground lighting information during a video conference.
[0065] The participant detection tool 502 receives a participant video stream from participant device, such as the participant device 406 in FIG. 4. The participant video stream includes a composite foreground and virtual background. The participant detection tool 502 detects a participant from the participant video stream.);
determine a beam direction of a lighting source illuminating the user in the
video stream (Nguyen, [0068] In some implementations, the lighting level adjustment
tool 506 can determine lighting directionality based on the foreground spatial distribution. For example, the foreground spatial distribution can indicate pixels or regions which are light or bright in contrast to pixels or regions which are dark. This can be used to indicate where light is coming from and in what direction. The lighting level
adjustment tool 506 can then adjust the virtual background lighting levels to follow the determined light direction.);
identify a first virtual background image presenting a background illuminated from a first direction that closely aligns with the beam direction of the
lighting source illuminating the user (Moon, [0025] After that, the control unit (130) can process and display a virtual object image to be displayed on the actual object and background illuminated by the light source unit (110) based on the confirmed three-dimensional information of the light source unit (110).
[0026] First, in step S110 of FIG. 1a, a light source unit (110) that moves by or with the user can receive beams from each of a plurality of optical transceivers (120) installed at different predetermined locations.
[0034] After that, in step S130, the control unit (130) can process a virtual object image to be displayed on the actual object and background illuminated by the light source unit (110) based on the result of verifying the three-dimensional information of the light source unit (110)); and
generate and present a composite video feed comprising a foreground image of the video stream superimposed on the first virtual background image, the composite video feed for transmission to the one or more second electronic devices during the video communication session (Nguyen, [0072] The adjusted video stream outputting tool 508 outputs the adjusted video stream along with instructions, commands, or other information configured to cause the device of the participant(s) to output the adjusted video stream to the participant(s) during the conference, where the adjusted video stream is produced from the adjusted virtual background and the foreground ( or adjusted foreground when appropriate).
[0091] At 808, the adjusted video stream is output. The adjusted video stream can be generated by overlaying the foreground (adjusted foreground when applicable) on the adjusted virtual background to produce a composite image, which is output as part of an adjusted video stream.).
Regarding Claim 2. The combination of Nguyen and Moon further teaches The electronic device of claim 1, wherein to generate the composite video feed, the processor is configured to cause the electronic device to:
extract a video image of the user from the video stream (Nguyen, [0065] The participant detection tool 502 receives a participant video stream from participant device, such as the participant device 406 in FIG. 4. The participant video stream includes a composite foreground and virtual background. The participant detection tool 502 detects a participant from the participant video stream. A region identified with the detected participant is denoted the foreground.); and
overlay the video image of the user on the first virtual background image (Nguyen, [0091] At 808, the adjusted video stream is output. The adjusted video stream can be generated by overlaying the foreground (adjusted foreground when applicable) on the adjusted virtual background to produce a composite image, which is output as part of an adjusted video stream.).
Claim 12 is similar in scope as Claim 1, and thus is rejected under same rationale.
Claim 19 is similar in scope as Claim 1, and thus is rejected under same rationale.
Claims 3-6, 8, 13-15, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen (US20230231972) in view of Moon (KR20200052074) further in view of Tangeland et al (US20220070389).
Regarding Claim 3. The combination of Nguyen and Moon fails to explicitly teach, however, Tangeland teaches The electronic device of claim 1, further comprising one or more output devices communicatively coupled to the processor and comprising a display, and wherein the processor is configured to cause the electronic device to:
generate and render, via the display, a video communication session window that comprises the composite video feed (Tangeland, abstract, the invention teaches a method for virtual background replacement during a video communication session. A frame comprising a user image and a captured background is obtained from a video capture device as part of a video stream. The frame is processed to replace the captured background with a virtual background. A change to an exposure characteristic of the video capture device is determined. In response to determining the change to the exposure characteristic, the virtual background is modified to match a virtual background brightness with a user image brightness to produce a modified virtual background. The frame is modified by combining the user image with the modified virtual background.
[0042] FIG. 2A depicts a video frame 200 before processing to match a foreground and a virtual background in accordance with an example embodiment. As depicted, frame 200 includes a foreground image 210 and a background image 220. Foreground image 210 is a user image that includes a user's person (head, face, body, etc.), which can be extracted using the image processing techniques described herein to insert the foreground image over a virtual background. Background image 220 depicts the actual environment of the user as captured by a camera, which is an indoor environment in the depicted example.).
Nguyen, Moon and Tangeland are analogous art because they all teach method of rendering virtual light in a composite video including user image. Tangeland further teaches GUI for displaying the composite video. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the virtual light rendering method for a composite video (taught in Nguyen and Moon), to further use the GUI window to display the composite video (taught in Tangeland), so as to provide a convenient user interface to replace virtual background and modify the composite video so that the background matches with the foreground (user image) (Tangeland, [0002]).
Regarding Claim 4. The combination of Nguyen, Moon and Tangeland further teaches The electronic device of claim 1, wherein the processor is configured to cause the electronic device to:
transmit, via the communications subsystem, the composite video feed to the one or more second electronic devices that are configured to generate and render, via a respective second display, a video communication session window containing the composite video feed (Tangeland, [0024] Client devices 102A-102N each include a network interface (I/F) 104, at least one processor 106, a display 108, a microphone 110, a speaker 112, a camera 114, and memory 116 that includes a client module 118, a camera module 120, a background module 122, and a matching module 124. Each client device 102A-102N may include a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, a video conferencing endpoint device, or any other programmable electronic device capable of executing computer readable program instructions. Network interface 104 enables each client device 102A-102Nto send and receive data over a network, such as network 140.
[0042] FIG. 2A depicts a video frame 200 before processing to match a foreground and a virtual background in accordance with an example embodiment. As depicted, frame 200 includes a foreground image 210 and a background image 220. Foreground image 210 is a user image that includes a user's person (head, face, body, etc.), which can be extracted using the image processing techniques described herein to insert the foreground image over a virtual background. Background image 220 depicts the actual environment of the user as captured by a camera, which is an indoor environment in the depicted example.).
The reasoning for combination of Nguyen, Moon and Tangeland is the same as described in Claim 3.
Regarding Claim 5. The combination of Nguyen, Moon and Tangeland further teaches The electronic device of claim 1, further comprising:
one or more output devices communicatively coupled to the processor; and
one or more input device communicatively coupled to the processor and comprising a display, and wherein the processor is configured to cause the electronic device (Tangeland, [0024] Client devices 102A-102N each include a network interface (I/F) 104, at least one processor 106, a display 108, a microphone 110, a speaker 112, a camera 114, and memory 116 that includes a client module 118, a camera module 120, a background module 122, and a matching module 124. Each client device 102A-102N may include a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, a video conferencing end point device, or any other programmable electronic device capable of executing computer readable program instructions. Network interface 104 enables each client device 102A-102Nto send and receive data over a network, such as network 140.) to:
present, via the display, a virtual background selection window containing the first virtual background image and a control feature that enables selection of the first virtual background image by the user via the one or more input device (Tangeland, [0039] Background recommendation module 136 may receive data, including image data captured by camera 114 of any of client devices 102A-102N and other data, in order to recommend one or more virtual backgrounds based on analysis of the received data. In particular, background recommendation module 136 may receive a data package that includes data such as a focal length of camera 114, image properties, scene data, and time of day and location data. Background recommendation module 136 may analyze the data included in the data package by processing a set of virtual background images to select a subset of virtual background images that best match the image properties and other data of the data package. Background recommendation module 136 may transmit the subset of virtual background images to the client device requesting a background recommendation, and a user may then make a selection of a background image in particular to utilize as a virtual background.).
The reasoning for combination of Nguyen, Moon and Tangeland is the same as described in Claim 3.
Regarding Claim 6. The combination of Nguyen, Moon and Tangeland further teaches The electronic device of claim 5, wherein, in presenting, via the display, the virtual background selection window containing the first virtual background image and the control, the processor is configured to cause the electronic device to:
indicate in the virtual background selection window that the first virtual background image has correct lighting; and
indicate in the virtual background selection window that the second virtual background image does not have correct lighting (Tangeland, [0019] Accordingly, embodiments are presented herein that enable the matching of foregrounds and virtual backgrounds during a video communication session. Image properties of both the foreground and the virtual background are analyzed to identify and perform image processing operations to either the foreground, the virtual background, or both, in order to match the foreground and the virtual background. The image properties that are considered can include any properties of the foreground and/or background, such as brightness, color, exposure, sharpness, contrast, white balance, noise, camera focal length, resolution, and the like.
[0039] Background recommendation module 136 may receive data, including image data captured by camera 114 of any of client devices 102A-102N and other data, in order to recommend one or more virtual backgrounds based on analysis of the received data. In particular, background recommendation module 136 may receive a data package that includes data such as a focal length of camera 114, image properties, scene data, and time of day and location data. Background recommendation module 136 may analyze the data included in the data package by processing a set of virtual background images to select a subset of virtual background images that best match the image properties and other data of the data package. Background recommendation module 136 may transmit the subset of virtual background images to the client device requesting a background recommendation, and a user may then make a selection of a background image in particular to utilize as a virtual background.
Therefore, the system detects the matching condition of the virtual background and foreground based on the image properties such as brightness. It is obvious to a user with ordinary skill in the art that, the system provides a set of virtual background for user to select, which is an indication that some degree of mismatch is detected. While when system doesn’t provide a set of virtual background for user to select, it is indicated that the background and foreground matches.).
The reasoning for combination of Nguyen, Moon and Tangeland is the same as described in Claim 3.
Regarding Claim 8. The combination of Nguyen, Moon and Tangeland further teaches The electronic device of claim 1, wherein the processor is configured to cause the electronic device to:
in response to determining that a currently selected virtual background image is not the first virtual background image, generate and render a virtual background notification window containing a recommendation for the user to select the first virtual background image (Tangeland, [0039] Background recommendation module 136 may receive data, including image data captured by camera 114 of any of client devices 102A-102N and other data, in order to recommend one or more virtual backgrounds based on analysis of the received data. In particular, background recommendation module 136 may receive a data package that includes data such as a focal length of camera 114, image properties, scene data, and time of day and location data. Background recommendation module 136 may analyze the data included in the data package by processing a set of virtual background images to select a subset of virtual background images that best match the image properties and other data of the data package. Background recommendation module 136 may transmit the subset of virtual background images to the client device requesting a background recommendation, and a user may then make a selection of a background image in particular to utilize as a virtual background.
[0043] FIG. 2B depicts a video frame 250 after processing to match a foreground and a virtual background in accordance with an example embodiment. After extracting user image 210 and selecting a virtual background 230, matching module 124 analyzes and adjusts foreground image 210 and/or virtual background 230 to match foreground image 210 to virtual background 230. For example, since virtual background 230 depicts a sunny, outdoor environment, user image 210 may require an adjustment to increase the brightness (not specifically shown in FIG. 2A due to the difficulty to show in a line drawing). Once foreground image 210 and virtual background 230 are adjusted to match, foreground image 210 and virtual background 230 are combined to produce video frame 250, which can be transmitted to a destination.).
Claim 13 is similar in scope as Claim 2, 3 & 4, and thus is rejected under same rationale.
Claim 14 is similar in scope as Claim 5, and thus is rejected under same rationale.
Claim 15 is similar in scope as Claim 6, and thus is rejected under same rationale.
Claim 20 is similar in scope as Claim 2, 3, 4 & 5, and thus is rejected under same rationale.
Claims 7, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen (US20230231972) in view of Moon (KR20200052074), Tangeland et al (US20220070389) further in view of Zhu et al (US20250014268)
Regarding Claim 7. The combination of Nguyen, Moon and Tangeland fails to explicitly teach, however, Zhu teaches The electronic device of claim 1, wherein the processor is configured to cause the electronic device to:
periodically determine an updated beam direction of a lighting source illuminating the user in the video stream; and
in response to determining that the updated beam direction has changed from the first direction to a third direction:
identify a third virtual background image presenting a background illuminated from a direction that closely aligns with the third beam direction of the lighting source illuminating the user (Zhu, abstract, the invention describes a shadow rendering method. The method includes: determining scene ranges corresponding to a plurality of first levels, each scene range being a first visible scene range of a virtual scene corresponding to a respective one of the plurality of first levels; for a respective one of the plurality of first levels: determining a first virtual item whose shadow is present in a first visible scene range corresponding to the first level; determining a first projection depth map corresponding to the first level based on the first virtual item, the first projection depth map being configured for describing distances between a light source and a plurality of first vertices on a surface of the first virtual item facing the light source; and rendering the shadow of the first virtual item in the first visible scene range based on the first projection depth map.
[0045] Generally, when shadows in a visible scene are updated, the shadows in the visible scene need to be re-rendered. For example, when a direction of a light source that illuminates the visible scene changes, the shadows in the visible scene are updated accordingly. In this case, the shadows in the visible scene need to be re-rendered. In a possible implementation, the visible scene and the shadows in the visible scene are updated together. For example, when the visible scene changes, because different virtual items exist in different visible scenes, the shadows in the visible scene also need to change. To distinguish between the visible scene before the shadow update and the visible scene after the shadow update, the visible scene after the shadow update is denoted as a first visible scene, and the visible scene before the shadow update is denoted as a second visible scene.
In order to detect change of the light bean direction, it is obvious to a person with ordinary skill in the art that the system has to check the light direction at a regular basis. For example, to detect the lighting direction through the day, the detection can be set to check for update every hour.); and
Nguyen, Moon, Tangeland and Zhu are analogous art because they all teach method of rendering virtual light in a composite video including user image. Zhu further detecting light direction change and update the virtual video accordingly. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the virtual light rendering method for a composite video (taught in Nguyen, Moon and Tangeland), to further detecting the lighting direction change (taught in Zhu), so as to properly render shadow of a virtual object in the virtual scene (Zhu, [0003-0004]).
The combination of Nguyen, Moon, Tangeland and Zhu further teaches generate and present a second composite video feed comprising a foreground image of the video stream superimposed on the third virtual background image, the second composite video feed for transmission to the one or more second electronic devices during the video communication session (Nguyen, [0065] The participant detection tool 502 receives a participant video stream from participant device, such as the participant device 406 in FIG. 4. The participant video stream includes a composite foreground and virtual background. The participant detection tool 502 detects a participant from the participant video stream.
[0072] The adjusted video stream outputting tool 508 outputs the adjusted video stream along with instructions, commands, or other information configured to cause the device of the participant(s) to output the adjusted video stream to the participant(s) during the conference, where the adjusted video stream is produced from the adjusted virtual background and the foreground ( or adjusted foreground when appropriate).
[0091] At 808, the adjusted video stream is output. The adjusted video stream can be generated by overlaying the foreground (adjusted foreground when applicable) on the adjusted virtual background to produce a composite image, which is output as part of an adjusted video stream.).
Claim 16 is similar in scope as Claim 7, and thus is rejected under same rationale.
Allowable Subject Matter
Claims 9-11, 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 9, it recites “…wherein the processor is configured to cause the electronic device to:
in response to determining that a change in selection of a currently selected virtual background image has not occurred within a period of time following presentation of the recommendation, change the currently selected virtual background image to the first virtual background image.” in the context of Claim 9.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 9. Therefore, Claim 9 is allowable over prior art.
Regarding Claim 10, it recites “…further comprising an artificial intelligence (Al) engine, and wherein the processor is configured to cause the electronic device to determine a lighting direction of one of a background image and of the user in the video feed by using the AI engine trained to determine a lighting direction of an image.” in the context of Claim 10.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 10. Therefore, Claim 10 is allowable over prior art.
Regarding Claim 11, it recites “…wherein the plurality of virtual background images comprises the first virtual background image and at least one second virtual background image that is a structurally identical background image to the first virtual background image and has a different light source direction from among the plurality of different directions.” in the context of Claim 11.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 11. Therefore, Claim 11 is allowable over prior art.
Claim 17 recites similar limitations as discussed above with regard to claim 8&9. Therefore, claim 17 is allowable over prior art.
Claim 18 recites similar limitations as discussed above with regard to claim 10. Therefore, claim 18 is allowable over prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Grabowski et al (US20090005961), abstract, the invention describes an en-route navigation system such as for a vehicle, an image of a cable is presented as a navigation object in a head-up display to indicate a route that the vehicle should follow. In particular embodiments the cable appears to an observer to be a real cable existing in the landscape and extending higher than the head of the observer as would, for example, a trolley cable. The cable is illustratively displayed volumetrically and with an optic flow that is consistent with the optic flow of the landscape when the vehicle is moving, thereby creating the impression that it is real. As a result, the cable can be displayed without any accompanying images that correlate points on the cable with locations in the landscape and yet nonetheless serve as a very useful tool for indicating to a driver the route over which the vehicle should go. The cable may be in any of a number of forms including a continuous line, a line with non-closely-spaced gaps, a line having non-closely-spaced segments that have a different luminance from the rest of the line or a string of closely spaced objects.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/ Primary Examiner, Art Unit 2619
Prosecution Insights