DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/30/2026 has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 1, 19-20 recite the limitation "… an object video … the object video … a positional relationship between the display device, the object, and the camera … detected depth of the object …" (bold emphasis added to accentuate insufficient antecedent basis). The limitation lacks clarity since “the object” has not been previously stated.
For the purposes of examination, the limitation is interpreted as the following:
“… a positional relationship between the display device, the object video, and the camera … detected depth of the object video …”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-7, 9, 7, 16, 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Swatko et al., hereinafter referred to as Swatko (US 9,681,041 B1) in view of Ocean et al., hereinafter referred to as Ocean et al., hereinafter referred to as Ocean (US 2022/0351751 A1).
As per claim 1, Swatko discloses an information processing device (Swatko: Abstract) comprising:
circuitry configured to
generate mask information (luma mask 66) for each frame of a captured video including an object video (video portion associated with subject 1 or object 61) and a display video (video portion associated with background 62), the captured video being captured by a camera (41), and
perform determination processing related to the object video (subject 1 or object 61) and the display video (background) in the captured video (motion picture 27), by using the mask information (luma mask 66) for separating the object video (subject 1 or object 61) and the display video (background) in the captured video (motion picture 27), obtained by capturing the object video and the display video displayed on a display device (LED backdrop 31),
wherein the circuitry performs the determination processing according to a positional relationship between the display device, the object, and the camera (Swatko: Figs. 1, 10, 20; Col. 9, ll. 64 – Col. 10, ll. 1-15; Col. 15, ll. 37-67; Col. 16, ll. 47-65 disclose a program-controlled unit 42 of camera 41 captures motion picture containing subject 1 object 61 along with a background 62 that represents LED backdrop 31 and applies a luma mask 66 for separating the subject and the background.), and
wherein the positional relationship indicates a detected depth of the display device and a detected depth of the object with respect to the camera.
However, Swatko does not explicitly disclose “… wherein the positional relationship indicates a detected depth of the display device and a detected depth of the object with respect to the camera.”.
Further, Ocean is in the same field of endeavor and teaches wherein the positional relationship indicates a detected depth of the display device and a detected depth of the object with respect to the camera (Ocean: Para. [0033] discloses "depth compositing tool that uses stereo cameras, LIDAR, and both stereo cameras and LIDAR to generate depth maps [claimed detected depth] … depth tool can be set to a particular depth range so that everything out of that depth range is discarded [claimed detected depth of the display device]"; Ocean: Para. [0074] discloses "makes use of a depth matte generated by the tracker … depth generated by stereo disparity of the tracker's stereo cameras"; Ocean: Para. [0076] discloses "Virtual depth occlusion uses the refined depth matte … to virtually occlude elements of the video camera video feed. This allows the software to place virtual objects that are closer to the camera in front of actors [claimed detected depth of the object with respect to the camera]").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko and Ocean before him or her, to modify the image processing system of Swatko to include the depth indicating positional relationship feature as described in Ocean. The motivation for doing so would have been to improve object detection tracking by providing a configuration that generates robust masking techniques.
As per claim 3, Swatko discloses the information processing device according to claim 1, wherein as the determination processing, the circuitry performs processing of determining whether a subject recognized by subject determination in the captured video is the object video or is the display video (Swatko: Figs. 1, 10, 20; Col. 16, ll. 47-65 disclose generating a luma mask 66, delineates the subject with the color black and the background with the color white).
As per claim 4, Swatko discloses the information processing device according to claim 1, wherein as the determination processing, the circuitry is further configured to determine, in the captured video, the object video area (foreground) in which the object video is shown or the display video area (background) in which the display video is shown, and perform subject determination in at least one of the object video area or the display video area (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65 disclose generating a luma mask 66, delineates the subject with the color black [performs subject determination] and the background with the color white).
As per claim 5, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to control the camera to capture the display video of the display device and the object, based on a result of the determination processing (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects with an LED backdrop via programmed control 42).
As per claim 6, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform control related to display of a subject on the camera that captures the display video of the display device and the object, based on a result of the determination processing (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects and a subject with an LED backdrop via programmed control 42 as shown in figure 20).
As per claim 7, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform control related to a focusing operation on the camera that captures the display video of the display device and the object, based on a result of the determination processing (Swatko: Figs. 1, 10, 20; Col. 11, ll. 22-28; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose using a selected preview window [focusing operation], the contents of which are to be included in one or more photographs to be taken, the one or more subjects define, e.g., using a camera height/centering control such as "UP" and "DOWN" buttons 25, a plurality of objects to be included in the window for the one or more photographs to be taken).
As per claim 9, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform control related to the display video of the display device based on a result of the determination processing (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects with an LED backdrop via programmed control 42).
As per claim 16, Swatko discloses the information processing device according to claim 1, wherein the circuitry generates the mask information for each respective frame of the captured video at a time of imaging, and performs the determination processing in the frame (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose as shown in figure 20, the luma mask 66 is generated for each frame of the captured video).
As per claims 19-20, the claim(s) recites analogous limitations to claim(s) 1 above, and is/are therefore rejected on the same premise.
As per claim 21, Swatko-Ocean disclose the information processing device according to claim 1, wherein the detected depth of the display device and the detected depth of the object are detected using subject light obtained on a same optical axis as used by the camera to capture the captured video (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects and a subject with an LED backdrop 31 via programmed control 42 displaying the background video on a background 62 and Ocean: Paras. [0033], [0063], [0074], [0076] disclose using a depth compositing tool that uses stereo cameras, LIDAR, and both stereo cameras and LIDAR to generate depth maps of actors.).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Swatko in view of Ocean in further view of Dsouza et al., hereinafter referred to as Dsouza (US 2019/0355172 A1).
As per claim 2, Swatko discloses the information processing device according claim 1, wherein a display video displayed on the display device is a background video (31) displaying the background video on a background (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects and a subject with an LED backdrop 31 via programmed control 42 displaying the background video on a background 62).
However, Swatko-Ocean do not explicitly disclose “… a background video obtained by rendering 3D background data …”.
Further, Dsouza is in the same field of endeavor and teaches a background video obtained by rendering 3D background data (Dsouza: Paras. [0036], [0055] disclose system 200 generating the background image in 3D).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Dsouza before him or her, to modify the image processing system of Swatko-Ocean to include the 3D background data feature as described in Dsouza. The motivation for doing so would have been to improve user experience by providing a configuration that allows the system to adapt to orientation, position, or movement of a camera.
Claims 8, 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Swatko in view of Ocean in further view of Watson et al., hereinafter referred to as Watson (US 2021/0287385 A1).
As per claim 8, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform (Swatko: Figs. 1, 10, 20; Col. 11, ll. 22-28; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose using a selected preview window [focusing operation], the contents of which are to be included in one or more photographs to be taken, the one or more subjects define, e.g., using a camera height/centering control such as "UP" and "DOWN" buttons 25, a plurality of objects to be included in the window for the one or more photographs to be taken).
However, Swatko-Ocean do not explicitly disclose “… exposure control …”.
Further, Watson is in the same field of endeavor and teaches performing exposure control (Watson: Paras. [0029], [0032], [0061] disclose camera assembly 125 captures image data that can be appended with metadata describing other details of the image data including sensory data (e.g. temperature, brightness of environment) or capture data (e.g. exposure, warmth, shutter speed, focal length, capture time, etc.) and adjusts its parameters when taking various images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Watson before him or her, to modify the camera system of Swatko-Ocean to include the exposure control feature as described in Watson. The motivation for doing so would have been to improve user experience by providing algorithms that enable feedback regarding geometric properties used to capture image data.
As per claim 12, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to generate (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44).
However, Swatko-Ocean do not explicitly disclose “… metadata to be associated with the captured video …”.
Further, Watson is in the same field of endeavor and teaches metadata to be associated with the captured video (Watson: Paras. [0029], [0032], [0061] disclose camera assembly 125 captures image data that can be appended with metadata describing other details of the image data including sensory data (e.g. temperature, brightness of environment) or capture data (e.g. exposure, warmth, shutter speed, focal length, capture time, etc.) and adjusts its parameters when taking various images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Watson before him or her, to modify the camera system of Swatko-Ocean to include the metadata feature as described in Watson. The motivation for doing so would have been to improve user experience by providing algorithms that enable feedback regarding geometric properties used to capture image data.
As per claim 13, Swatko discloses the information processing device according to claim 12, (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44).
However, Swatko-Ocean do not explicitly disclose “… wherein the metadata includes a determination result …”.
Further, Watson is in the same field of endeavor and teaches wherein the metadata includes a determination result (Watson: Paras. [0029], [0032], [0061] disclose camera assembly 125 captures image data that can be appended with metadata describing other details of the image data including sensory data (e.g. temperature, brightness of environment) or capture data (e.g. exposure, warmth, shutter speed, focal length, capture time, etc.) and adjusts its parameters when taking various images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Watson before him or her, to modify the camera system of Swatko-Ocean to include the metadata feature as described in Watson. The motivation for doing so would have been to improve user experience by providing algorithms that enable feedback regarding geometric properties used to capture image data.
As per claim 14, Swatko discloses the information processing device according to claim 12, (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44).
However, Swatko-Ocean do not explicitly disclose “… wherein the metadata includes a control parameter …”.
Further, Watson is in the same field of endeavor and teaches wherein the metadata includes a control parameter (Watson: Paras. [0029], [0032], [0061] disclose camera assembly 125 captures image data that can be appended with metadata describing other details of the image data including sensory data (e.g. temperature, brightness of environment) or capture data (e.g. exposure, warmth, shutter speed, focal length, capture time, etc.) and adjusts its parameters when taking various images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Watson before him or her, to modify the camera system of Swatko-Ocean to include the metadata feature as described in Watson. The motivation for doing so would have been to improve user experience by providing algorithms that enable feedback regarding geometric properties used to capture image data.
As per claim 15, Swatko-Watson disclose the information processing device according to claim 12, wherein the metadata includes the mask information (Watson: Paras. [0029], [0032], [0060]-[0061] disclose camera assembly 125 includes two cameras and is configured to capture stereo image data that can be appended with metadata describing other details of the image data including binary segmentation masks).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Swatko in view of Ocean in further view of Chu et al., hereinafter referred to as Chu (US 2022/0256116 A1).
As per claim 10, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform control related to (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects with an LED backdrop via programmed control 42).
However, Swatko-Ocean do not explicitly disclose “… a focus state of a display video …”.
Further, Chu is in the same field of endeavor and teaches a focus state of a display video (Chu: Paras. [0070], [0078] disclose analyzing the video data to determine boundaries between the foreground portion 501 and the background portion 503 based on the difference in focus therebetween, which is derived from adjusting an effective aperture of the lens 204 to change an actual depth of field of the camera device to the desired depth of field 314 a, b).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Chu before him or her, to modify the camera system of Swatko-Ocean to include the focus state feature as described in Chu. The motivation for doing so would have been to improve composition of the video stream by providing additional integrated hardware and software to differentiate between desired and undesired portions of a video stream.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Swatko in view of Ocean in further view of Gleich (DE-102013209323-A1).
As per claim 11, Swatko discloses the information processing device according to claim 1, wherein the circuitry is further configured to perform control related to (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects with an LED backdrop via programmed control 42).
However, Swatko-Ocean do not explicitly disclose “… performs control related to luminance of a display video …”.
Further, Gleich is in the same field of endeavor and teaches performs control related to luminance of a display video (Gleich: Figs. 1-2 & Para. [0041] disclose controlling brightness of the partial regions of the LED wall region 12).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Gleich before him or her, to modify the LED camera configuration of Swatko-Ocean to include the luminance control feature as described in Gleich. The motivation for doing so would have been to improve image capture quality by providing techniques to prevent disruptive reflections onto the foreground.
Claim 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Swatko in view of Ocean in view of Wan et al., hereinafter referred to as Wan (US 2017/0370702 A1) in further view of Shaddix et al., hereinafter referred to as Shaddix (US 2021/0064849 A1).
As per claim 17, Swatko discloses the information processing device according to claim 1, wherein the mask information is generated based on a video obtained by
However, Swatko-Ocean do not explicitly disclose “… an infrared short wavelength camera that captures a video having a same visual field range as the captured video.”
Further, Wan is in the same field of endeavor and teaches an infrared (Wan: Fig. 2 & Para. [0021] disclose an infrared wavelength camera 205 that captures a same video as the captured video).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean and Wan before him or her, to modify the camera system of Swatko-Ocean to include the infrared camera feature as described in Wan. The motivation for doing so would have been to improve object recognition functions by providing a multi-functional camera that integrates advanced image capturing components.
However, Swatko-Ocean-Wan do not explicitly disclose “… an infrared short wavelength camera …”
Furthermore, Shaddix is in the same field of endeavor and teaches an infrared short wavelength camera (Shaddix: Para. [0034] discloses camera system 120 including SWIR sensors 122).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Swatko-Ocean-Wan and Shaddix before him or her, to modify the camera configuration of Swatko-Ocean-Wan to include the infrared short wavelength camera feature as described in Shaddix. The motivation for doing so would have been to improve signal-to-noise ratio associated with binary mask images by providing additional threshold conditions during image capture.
As per claim 18, Swatko-Ocean-Wan-Shaddix disclose the information processing device according to claim 17, wherein the infrared short wavelength camera is configured in such a manner that subject light is incident on a same optical axis as the camera that obtains the captured video obtained by capturing the display video and an object (Swatko: Figs. 1, 10, 20; Col. 15, ll. 37-67; Col. 16, ll. 47-65; Col. 18, ll. 37-44 disclose the camera captures objects with an LED backdrop via programmed control 42 and Wan: Fig. 2 & Para. [0021] disclose an infrared wavelength camera 205 that is configured in such a manner that subject light is incident on a same optical axis as a visible camera 206 that obtains the same scene and Shaddix: Para. [0034] discloses camera system 120 including SWIR sensors 122).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and can be viewed in the list of references.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEET DHILLON whose telephone number is (571)270-5647. The examiner can normally be reached M-F: 5am-1:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sath V. Perungavoor can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEET DHILLON/Primary Examiner
Art Unit: 2488
Date: 04-04-2026