DETAILED ACTION
Response to Amendment
This Action is responsive to Applicant’s response filed on 03/04/2026. All claims are still pending
in the present application. This Action is made FINAL.
Amendment
Applicant submitted amendments on 03/04/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Examiner’s Responses:
Applicant’s arguments, see Remarks, filed 03/04/2026, with respect to the rejection(s) of claim(s) 1-3, 9-11, 14, and 16-18 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn.
However, Applicant’s arguments have been considered but are moot in view of the new ground(s) of rejection in view of Lee et al (US 2019/0311202 A1), Yang et al (US 2015/0091900 A1), and Yi et al (US 2021/0150678 A1).
Office Action Summary
Claim(s) 1-5, 9-14, and 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 2019/0311202 A1) in view of Yang et al (US 2015/0091900 A1), further in view of Yi et al (US 2021/0150678 A1).
Claim(s) 6-8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 2019/0311202 A1) in view of Yang et al (US 2015/0091900 A1) and Yi et al (US 2021/0150678 A1), further in view of Agarwal et al (US 8,125,510 B2).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 9-14, and 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 2019/0311202 A1) in view of Yang et al (US 2015/0091900 A1), further in view of Yi et al (US 2021/0150678 A1).
Regarding claim(s) 1, Lee teaches a method, comprising:
receiving a first video stream (Figure 1A; Figure 1B; and Paragraph [0042]: “video stream 100 includes a set of n video frames 110-1, 110-2, 110-3, . . . , and 110-n (collectively video frames 110) that are sequential in time”);
analyzing a video frame of the first video stream to identify a region of interest (read as “target object”) of the video frame (Figure 1A; Figure 1B; Figure 12: Receive an image and a corresponding object mask identifying an object in the image 1210; Paragraph [0042]: “Each video frame 110 includes a foreground object 120 (e.g., a car) to be segmented from the background in each video frame 110”; Paragraph [0043]: “In general, the first segmentation mask 150-1 for the first video frame 110-1 is given or otherwise annotated before segmenting video stream 100, such that it is known which target object is to be segmented in the video frame”; and Paragraph [0060]: “encoder-decoder network 550 takes inputs of a target frame 510 and an estimated mask 520 of the previous frame, and a reference frame 530 and a ground-truth mask 540 of reference frame 530, and outputs an estimated mask 560 for target frame 510. Reference frame 530 and ground-truth mask 540 of reference frame 530 can help to detect a target object in target frame 510, and estimated mask 520 of the previous frame can be propagated to target frame 510 to estimate mask 560 for target frame 510”);
performing object detection on the region of interest to identify a region comprising a representation of an object (Figure 5; and Paragraph [0060]: “As illustrated, an encoder-decoder network 550 takes inputs of a target frame 510 […] Reference frame 530 and ground-truth mask 540 of reference frame 530 can help to detect a target object in target frame 510, and estimated mask 520 of the previous frame can be propagated to target frame 510 to estimate mask 560 for target frame 510”); and
generating a second video stream comprising the modified image (Paragraph [0059]: “a neural network including two encoders (e.g., a Siamese encoder network) is used to both detect a target object in a video stream […]” and “The fine-tuned network can be used to segment any video stream using a reference frame (e.g., the first frame) of the video stream and a corresponding ground-truth segmentation mask […]”; Paragraph [0061]: “Second encoder 630 takes an input 615, which includes a target video frame in a video stream and an estimated mask of the previous video frame in the video stream, and extracts a feature map 635 from input 615”; and Claim 1: “accessing […] a target frame in the video stream […] extracting […] a target segmentation mask for the target frame from the combined feature map”).
Lee fails to teach to performing object detection on the region of interest to identify a region comprising a representation of an object that is detected to be a distorted representation of the object; masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object; and replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object with an undistorted representation of the object to generate a modified image.
However, Yang teaches to performing object detection on the region of interest to identify a region comprising a representation of an object that is detected to be a distorted representation of the object (Paragraph [0003]: “Close range portraiture photographs, such as self-portraits, are often perceived as having apparent perspective distortions at typical image viewing distances […] tends to magnify the size of the nose and chin, among other features”; Paragraph [0004]: “faces are automatically detected and segmented […] create a new image in which apparent perspective distortion is reduced”; and Paragraph [0005]: “detect an object within the image data and a distance from the initial viewpoint to the object […]”);
(Figure 3B; Paragraph [0037]: “A warp (306) can be applied to the face image data based upon the depth map to rerender the face from a more distant viewpoint, thereby correcting for perspective distortion”; Paragraph [0039]: “The rerendered face image data and the inpainted background image data can be composited (310) to generate a new image in which perspective distortion is eliminated”; and Paragraph [0040]: “Accordingly, corrected image data and corresponding depth map image data (334) are output”).
Lee teaches receiving a first video stream, analyzing a video frame to identify a region of interest, performing object detection to identify a region comprising a representation of an object, and processing the object within a neural-network-based video-stream framework. Yang teaches that faces are automatically detected and segmented, that close range portrait images may exhibit apparent perspective distortion, and that the detected object image data may be rerendered from a synthetic viewpoint to generate perspective distortion corrected image data. Under the broadest reasonable interpretation, Yang's detected object image data corresponds to the claimed distorted representation of the object, while Yang's rerendered and perspective distortion corrected image data corresponds to the claimed undistorted representation of the object.
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Lee's video-stream-based object detection framework to incorporate Yang's distortion-correction techniques in order to identify distorted object representations and generate corrected representations thereof, because Yang teaches reducing or eliminating perspective distortion associated with detected objects, thereby improving the visual accuracy and realism of the object representation. The motivation for this combination of references would have been to predictably resulted in a system that detects an object within a video frame, identifies a distorted representation of the object, generates an undistorted representation of the same object, and processes the resulting image within Lee's video-stream processing environment. This motivation for the combination of Lee and Yang is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Lee and Yang fail to teach to masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object; and replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object
However, Yi teaches to masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object (Figure 8; Paragraph [0008]: “obtaining an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted; down-sampling the original high-resolution image to obtain a low-resolution image to be inpainted; generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image”; Paragraph [0074]: “The portion(s) of the original high-resolution image to be inpainted may be referred to herein as the inside-mask area, and the portion(s) of the original high-resolution image that is not inpainted may be referred to herein as the outside-mask area”; and Paragraph [0136]: “At 802, an original high-resolution image […] is received to be inpainted. An inpainting mask is also received[…] The inpainting mask may be defined by the user […]”); and
replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object (Figure 8; Paragraph [0008]: “generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image […]”; Paragraph [0138]: “At 806, the trained inpainting generator 101 is used to generate a low-resolution inpainted image […] Input to the trained inpainting generator 101 is the low-resolution image and the inpainting mask […]”; Paragraph [0140]: “generate an aggregated high-frequency residual image, which contains high-frequency residual information for at least the inside-mask area of the high-resolution image […] Then, the high-frequency residual of each inside-mask region of the residual image is calculated”; and Paragraph [0142]: “the aggregated high-frequency residual image is combined with a low-frequency inpainted image generated from the low-resolution inpainted image generated at step 806”).
Lee teaches receiving a first video stream, analyzing a video frame of the video stream, identifying a region of interest corresponding to a target object, performing object detection to identify a region comprising a representation of the object, and processing the object within a neural-network-based video-stream framework. Yang teaches that close range images may exhibit apparent perspective distortion, that faces are automatically detected and segmented, and that detected object image data may be rerendered from a synthetic viewpoint to generate perspective distortion corrected image data. Under the broadest reasonable interpretation, Yang's detected object image data corresponds to the claimed distorted representation of the object, while Yang's rerendered and perspective distortion corrected image data corresponds to the claimed undistorted representation of the object and modified image. Furthermore, Yi teaches receiving an inpainting mask identifying an inside-mask area, generating image content for the inside-mask area using a trained inpainting generator, and outputting a high-resolution inpainted image. The inpainting mask corresponds to the claimed masked region and the trained inpainting generator corresponds to the claimed machine learning model used to replace the masked region.
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Lee's video-stream-based object detection and segmentation system with Yang's distortion-correction techniques in order to identify distorted object representations and generate corrected representations thereof, and further to incorporate Yi's machine-learning-based inpainting techniques to mask and reconstruct selected object regions, because Yang teaches correcting or eliminating perspective distortion associated with detected objects (Paragraph [0003]-[0004] and Paragraph [0037]-[0040]) and Yi teaches reconstructing masked image regions using a trained machine-learning model (Paragraph [0008], [0138], and [0142]), yielding the predictable result of identifying an object within a video frame, determining that the object is represented in a distorted manner, masking the distorted representation, replacing the masked region using a machine learning model with an undistorted representation of the object, generating a modified image, and outputting the modified image within Lee's video-stream processing framework. This motivation for the combination of Lee, Yang, and Yi is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2, 10, and 17, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 9, wherein:
where Yi teaches replacing the masked region (Figure 8; Paragraph [0008]: “generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image […]”; Paragraph [0138]: “At 806, the trained inpainting generator 101 is used to generate a low-resolution inpainted image […] Input to the trained inpainting generator 101 is the low-resolution image and the inpainting mask […]”; Paragraph [0140]: “generate an aggregated high-frequency residual image, which contains high-frequency residual information for at least the inside-mask area of the high-resolution image […] Then, the high-frequency residual of each inside-mask region of the residual image is calculated”; and Paragraph [0142]: “the aggregated high-frequency residual image is combined with a low-frequency inpainted image generated from the low-resolution inpainted image generated at step 806”) with where Yang teaches the undistorted representation of the object is performed based upon the distorted representation of the object (Figure 3B; Paragraph [0037]: “A warp (306) can be applied to the face image data based upon the depth map to rerender the face from a more distant viewpoint, thereby correcting for perspective distortion”; Paragraph [0039]: “The rerendered face image data and the inpainted background image data can be composited (310) to generate a new image in which perspective distortion is eliminated”; and Paragraph [0040]: “Accordingly, corrected image data and corresponding depth map image data (334) are output”).
Regarding claim(s) 3, 11, 18, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 9, where Yi teaches wherein:
the machine learning model comprises a neural network model (Paragraph [0033]: “a neural network to be trained using lower resolution images, and the trained neural network may then be used for inpainting of a high-resolution image. Thus, the disclosed methods and systems provide the technical effect that a high-resolution image can be modified, by the removal or repositioning of an object in the image and/or to reconstruct a missing portion,”).
Regarding claim(s) 4, 12, and 19, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 9, where Yi teaches the operations comprising:
training the machine learning model using a second machine learning model (Paragraph [0114] – Paragraph [0115]: “Training of the inpainting generator 101 is now discussed. The inpainting generator 101 is trained using a discriminator 418 and a loss computation operation 420 […] training is performed by splitting the training objective into adversarial loss and reconstruction loss […]”; and Paragraph [0127] – Paragraph [0129]: “At step 610, the inpainting generator 101 is trained until the inpainting generator 101 converges […] Convergence may be checked for each training iteration, for example by calculating loss gradient or by calculating the weight gradient, and comparing against a defined convergence threshold […] Each training iteration for the inpainting generator 101 may be performed using steps 612-616, for example […] For consistency, the training data for the discriminator 418 and the training data for the inpainting generator 101 may be sampled from the same database and using the same sampling method”).
Regarding claim(s) 5, 13, and 20, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 12, wherein training the machine learning model comprises:
replacing, using the machine learning model and an object reference image associated with a second object, a second masked region of a second masked image with a representation of the second object to generate a second image (Figure 8; Paragraph [0008]: “obtaining an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted; down-sampling the original high-resolution image to obtain a low-resolution image to be inpainted; generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image”; Paragraph [0074]: “The portion(s) of the original high-resolution image to be inpainted may be referred to herein as the inside-mask area, and the portion(s) of the original high-resolution image that is not inpainted may be referred to herein as the outside-mask area”; and Paragraph [0136]: “At 802, an original high-resolution image […] is received to be inpainted. An inpainting mask is also received[…] The inpainting mask may be defined by the user […]”);
determining, using the second machine learning model, a context score associated with the second image and training the machine learning model based upon the context score (Paragraph [0114] – Paragraph [0115]: “Training of the inpainting generator 101 is now discussed. The inpainting generator 101 is trained using a discriminator 418 and a loss computation operation 420 […] training is performed by splitting the training objective into adversarial loss and reconstruction loss […]”; and Paragraph [0127] – Paragraph [0129]: “At step 610, the inpainting generator 101 is trained until the inpainting generator 101 converges […] Convergence may be checked for each training iteration, for example by calculating loss gradient or by calculating the weight gradient, and comparing against a defined convergence threshold […] Each training iteration for the inpainting generator 101 may be performed using steps 612-616, for example […] For consistency, the training data for the discriminator 418 and the training data for the inpainting generator 101 may be sampled from the same database and using the same sampling method”).
Regarding claim(s) 9 and 16, Lee teaches a non-transitory computer-readable medium storing instructions (Figure 19; and Paragraph [0124]) that when executed perform operations comprising:
identifying an image (Figure 1A; Figure 1B; Figure 12: Receive an image and a corresponding object mask identifying an object in the image 1210; Paragraph [0042]: “Each video frame 110 includes a foreground object 120 (e.g., a car) to be segmented from the background in each video frame 110”; Paragraph [0043]: “In general, the first segmentation mask 150-1 for the first video frame 110-1 is given or otherwise annotated before segmenting video stream 100, such that it is known which target object is to be segmented in the video frame”; and Paragraph [0060]: “encoder-decoder network 550 takes inputs of a target frame 510 and an estimated mask 520 of the previous frame, and a reference frame 530 and a ground-truth mask 540 of reference frame 530, and outputs an estimated mask 560 for target frame 510. Reference frame 530 and ground-truth mask 540 of reference frame 530 can help to detect a target object in target frame 510, and estimated mask 520 of the previous frame can be propagated to target frame 510 to estimate mask 560 for target frame 510”);
performing object detection on the region of interest to identify a region comprising a representation of an object (Figure 5; and Paragraph [0060]: “As illustrated, an encoder-decoder network 550 takes inputs of a target frame 510 […] Reference frame 530 and ground-truth mask 540 of reference frame 530 can help to detect a target object in target frame 510, and estimated mask 520 of the previous frame can be propagated to target frame 510 to estimate mask 560 for target frame 510”).
Lee fails to teach to performing object detection on the region of interest to identify a region comprising a representation of an object that is detected to be a distorted representation of the object; masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object; and replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object with an undistorted representation of the object to generate a modified image.
However, Yang teaches to performing object detection on the region of interest to identify a region comprising a representation of an object that is detected to be a distorted representation of the object (Paragraph [0003]: “Close range portraiture photographs, such as self-portraits, are often perceived as having apparent perspective distortions at typical image viewing distances […] tends to magnify the size of the nose and chin, among other features”; Paragraph [0004]: “faces are automatically detected and segmented […] create a new image in which apparent perspective distortion is reduced”; and Paragraph [0005]: “detect an object within the image data and a distance from the initial viewpoint to the object […]”);
(Figure 3B; Paragraph [0037]: “A warp (306) can be applied to the face image data based upon the depth map to rerender the face from a more distant viewpoint, thereby correcting for perspective distortion”; Paragraph [0039]: “The rerendered face image data and the inpainted background image data can be composited (310) to generate a new image in which perspective distortion is eliminated”; and Paragraph [0040]: “Accordingly, corrected image data and corresponding depth map image data (334) are output”).
Lee teaches receiving a first video stream, analyzing a video frame to identify a region of interest, performing object detection to identify a region comprising a representation of an object, and processing the object within a neural-network-based video-stream framework. Yang teaches that faces are automatically detected and segmented, that close range portrait images may exhibit apparent perspective distortion, and that the detected object image data may be rerendered from a synthetic viewpoint to generate perspective distortion corrected image data. Under the broadest reasonable interpretation, Yang's detected object image data corresponds to the claimed distorted representation of the object, while Yang's rerendered and perspective distortion corrected image data corresponds to the claimed undistorted representation of the object.
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Lee's video-stream-based object detection framework to incorporate Yang's distortion-correction techniques in order to identify distorted object representations and generate corrected representations thereof, because Yang teaches reducing or eliminating perspective distortion associated with detected objects, thereby improving the visual accuracy and realism of the object representation. The motivation for this combination of references would have been to predictably resulted in a system that detects an object within a video frame, identifies a distorted representation of the object, generates an undistorted representation of the same object, and processes the resulting image within Lee's video-stream processing environment. This motivation for the combination of Lee and Yang is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Lee and Yang fail to teach to masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object; and replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object
However, Yi teaches to masking the region comprising the distorted representation of the object to generate a masked image comprising a masked region corresponding to the object in place of the distorted representation of the object (Figure 8; Paragraph [0008]: “obtaining an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted; down-sampling the original high-resolution image to obtain a low-resolution image to be inpainted; generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image”; Paragraph [0074]: “The portion(s) of the original high-resolution image to be inpainted may be referred to herein as the inside-mask area, and the portion(s) of the original high-resolution image that is not inpainted may be referred to herein as the outside-mask area”; and Paragraph [0136]: “At 802, an original high-resolution image […] is received to be inpainted. An inpainting mask is also received[…] The inpainting mask may be defined by the user […]”); and
replacing, using a machine learning model, the masked region that replaced of the distorted representation of the object (Figure 8; Paragraph [0008]: “generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image […]”; Paragraph [0138]: “At 806, the trained inpainting generator 101 is used to generate a low-resolution inpainted image […] Input to the trained inpainting generator 101 is the low-resolution image and the inpainting mask […]”; Paragraph [0140]: “generate an aggregated high-frequency residual image, which contains high-frequency residual information for at least the inside-mask area of the high-resolution image […] Then, the high-frequency residual of each inside-mask region of the residual image is calculated”; and Paragraph [0142]: “the aggregated high-frequency residual image is combined with a low-frequency inpainted image generated from the low-resolution inpainted image generated at step 806”).
Lee teaches receiving a first video stream, analyzing a video frame of the video stream, identifying a region of interest corresponding to a target object, performing object detection to identify a region comprising a representation of the object, and processing the object within a neural-network-based video-stream framework. Yang teaches that close range images may exhibit apparent perspective distortion, that faces are automatically detected and segmented, and that detected object image data may be rerendered from a synthetic viewpoint to generate perspective distortion corrected image data. Under the broadest reasonable interpretation, Yang's detected object image data corresponds to the claimed distorted representation of the object, while Yang's rerendered and perspective distortion corrected image data corresponds to the claimed undistorted representation of the object and modified image. Furthermore, Yi teaches receiving an inpainting mask identifying an inside-mask area, generating image content for the inside-mask area using a trained inpainting generator, and outputting a high-resolution inpainted image. The inpainting mask corresponds to the claimed masked region and the trained inpainting generator corresponds to the claimed machine learning model used to replace the masked region.
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Lee's video-stream-based object detection and segmentation system with Yang's distortion-correction techniques in order to identify distorted object representations and generate corrected representations thereof, and further to incorporate Yi's machine-learning-based inpainting techniques to mask and reconstruct selected object regions, because Yang teaches correcting or eliminating perspective distortion associated with detected objects (Paragraph [0003]-[0004] and Paragraph [0037]-[0040]) and Yi teaches reconstructing masked image regions using a trained machine-learning model (Paragraph [0008], [0138], and [0142]), yielding the predictable result of identifying an object within a video frame, determining that the object is represented in a distorted manner, masking the distorted representation, replacing the masked region using a machine learning model with an undistorted representation of the object, generating a modified image, and outputting the modified image within Lee's video-stream processing framework. This motivation for the combination of Lee, Yang, and Yi is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 14, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 9, where Lee teaches the operations comprising:
analyzing the image to identify a region of interest (read as “target object”) of the image, wherein the object detection is performed on the region of interest (Figure 1A; Figure 1B; Figure 12: Receive an image and a corresponding object mask identifying an object in the image 1210; Paragraph [0042]: “Each video frame 110 includes a foreground object 120 (e.g., a car) to be segmented from the background in each video frame 110”; Paragraph [0043]: “In general, the first segmentation mask 150-1 for the first video frame 110-1 is given or otherwise annotated before segmenting video stream 100, such that it is known which target object is to be segmented in the video frame”; and Paragraph [0060]: “encoder-decoder network 550 takes inputs of a target frame 510 and an estimated mask 520 of the previous frame, and a reference frame 530 and a ground-truth mask 540 of reference frame 530, and outputs an estimated mask 560 for target frame 510. Reference frame 530 and ground-truth mask 540 of reference frame 530 can help to detect a target object in target frame 510, and estimated mask 520 of the previous frame can be propagated to target frame 510 to estimate mask 560 for target frame 510”).
Claim(s) 6-8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US 2019/0311202 A1) in view of Yang et al (US 2015/0091900 A1) and Yi et al (US 2021/0150678 A1), further in view of Agarwal et al (US 8,125,510 B2).
Regarding claim(s) 6 and 15, Lee as modified by Yang and Yi teaches the non-transitory computer-readable medium of claim 14, but do not specifically teach the region of interest comprises a representation of a desktop of a desk and the object corresponds to an item on the desktop.
However, Agarwal teaches the region of interest comprises a representation of a desktop of a desk and the object corresponds to an item on the desktop (Col. 3, lines 36-41: “At each user location a video camera 205, 206 is provided which captures video images of a user's tablet PC display as well as any objects in a region 207, 208 around that display visible to the camera, such as the user's hand, stylus, coffee mug, pens, or other objects”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Lee, Yang and Yi and Agarwal before the effective filing date of the claimed invention. The motivation for this combination of references would have been to a workspace surface as the region of interest in Yi’s image processing pipeline because Yi is agnostic to the semantic content of the region, and using a real-world surface such as a desktop with objects on it as the target region would have been a predictable, common application of Yi’s masking and inpainting framework. This motivation for the combination of Lee, Yang and Yi and Agarwal is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 7, Lee as modified by Yang and Yi teaches the method of claim 1, but do not specifically teach comprising: displaying the second video stream on a client device. However, Agarwal teaches to displaying the second video stream on a client device (Col. 3, lines 61-67: “The processor takes the foreground image and the image of the shared work product 56 and composes (block 54) an output image 55 by combining those images in any suitable manner. Examples of methods of combining that may be used are described below. The composite output image 55 may then be presented at a remote user display”; and Col. 6, lines 12-15: “a second display 404 for displaying a video conference image of one or more remote users. A second video camera 400 may optionally be provided to capture a stream of video images of a user at the location”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Lee, Yang and Yi and Agarwal before the effective filing date of the claimed invention. The motivation for this combination of references would have been to display Yi’s modified or inpainted video stream on a client device in the same manner as Agarwal’s remote display because both references concern processed visual content intended for user consumption. Using known remote-display mechanisms from Agarwal to present Yi’s inpainted video stream would have been a predictable application of established techniques. This motivation for the combination of Lee, Yang and Yi and Agarwal is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 8, Lee as modified by Yang and Yi teaches the method of claim 7, but do not specifically teach comprising: establishing a video call between the client device and a second client device, wherein: the first video stream is captured using a camera associated with the second client device; and displaying the second video stream on the client device is performed in response to establishing the video call.
However, Agarwal teaches to establishing a video call between the client device and a second client device (Col. 6, lines 11-15: “Optionally provided is a second display 404 for displaying a video conference image of one or more remote users. A second video camera 400 may optionally be provided to capture a stream of video images of a user at the location”), wherein:
the first video stream is captured using a camera associated with the second client device (Figure 4; and Col. 6, lines 4-15: “at a first user location, a display 403 is provided which may be a computer screen, tablet PC screen, or other suitable display […] Optionally provided is a second display 404 for displaying a video conference image of one or more remote users. A second video camera 400 may optionally be provided to capture a stream of video images of a user at the location.”); and
displaying the second video stream on the client device is performed in response to establishing the video call (Col. 3, lines 61-67: “The processor takes the foreground image and the image of the shared work product 56 and composes (block 54) an output image 55 by combining those images in any suitable manner. Examples of methods of combining that may be used are described below. The composite output image 55 may then be presented at a remote user display”; and Col. 6, lines 12-15: “a second display 404 for displaying a video conference image of one or more remote users. A second video camera 400 may optionally be provided to capture a stream of video images of a user at the location”).
Relevant Prior Art Directed to State of Art
Yadav et al (US 2021/0406589 A1) are relevant prior art not applied in the rejection(s) above. Yadav discloses a computer-implemented method comprising: receiving first image data representing a first image; selecting, from among a plurality of masker components, a first masker component configured to generate masked image data for classification processing; and processing the first image data using the first masker component, the processing comprising: determining, in the first image data, a first plurality of pixels as corresponding to a representation of a classification category, wherein the first plurality of pixels comprises a first pixel and a second pixel; determining, for the first pixel, first data indicating a first likelihood that the first pixel is relevant for the classification processing; determining, for the second pixel, second data indicating a second likelihood that the second pixel is relevant for the classification processing; determining, in the first image data, a second plurality of pixels are irrelevant for the classification processing; determining first pixel data to include the second plurality of pixels; determining, based on the second data, to include the second pixel in the first pixel data; determining second image data using the first pixel data and second pixel data associated with the first pixel, wherein the first pixel data is set to a value different than that corresponding to the second pixel and the second plurality of pixels; and processing the second image data, using a first task component, to determine classification data corresponding to the second image data.
Stec et al (US 2015/0262344 A1) are relevant prior art not applied in the rejection(s) above. Stec discloses an image acquisition system comprising: a first memory for storing at least a portion of a distorted input image acquired from an image sensor and a lens system; a second memory for storing a corrected output image; and an interpolator module connected to said first memory for reading distorted input image information and to said second memory for writing corrected output image information, said interpolator comprising: a bi-cubic interpolator; and a pair of bi-linear interpolators and being switchable between a first high quality mode and a second high speed mode where, in said first high quality mode, for each pixel for said output image, said interpolator is arranged to read a 4 x 4 pixel window from said first memory and with said bi-cubic interpolator to interpolate said 4 x 4 pixel window to provide said output pixel value, and in said second high speed mode, for each pair adjacent output pixels for said corrected output image, said interpolator is arranged to read a 4 x 4 pixel window from said first memory, said 4 x 4 pixel window bounding a pair of 2 x 2 pixel windows, each of which are interpolated in parallel by said pair of bi-linear interpolators to provide said output pixel values.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571)272-1361. The examiner can normally be reached M - F: 7:30am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Edward Urban can be reached on 571-272-7899. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights