DETAILED ACTION
This action is in response to the amendment filed 3/6/2026.
Claims 2-21 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 3/6/2026, regarding the rejection of claim 2 under 35 U.S.C. 103, have been fully considered but they are not persuasive.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant first argues on page 8 of the amendment, that Luan teaches that the business object is either in the foreground or background, but never temporarily in one or the other in different frames.
The Examiner respectfully disagrees.
Luan teaches in paragraph [0096] that a display image is to be displayed in multiple frames of a video of display images corresponding to the business object and determining a display position of the display image in the video and paragraphs [0100], where frames are selected sequentially for determination. Given the flowchart of Luan’s Fig. 3, this at least suggests that a determination as to whether a business object is overlapped on a frame by frame basis, and thus may “temporarily” be overlapped by a foreground image (see steps S308 and S310 of Fig. 3).
Applicant next describes the Gaddy reference on page 9 with no apparent argument, simply a statement of Applicant’s opinion of Gaddy’s functionality.
Applicant next argues on page 9 of the amendment that Marino enables a user to insert a reference box that would stay “in the foreground” and that Marino does not describe determining that content which was previously visible become occluded in a later frame.
The Examiner respectfully disagrees.
As Marino teaches in paragraph [0117], and corresponding Fig. 3A (reproduced below), a first frame of a video is shown, and in Fig. 3B (reproduced below), a second frame of the video is shown, wherein the second frame occurs “sometime after the first frame”.
PNG
media_image1.png
418
405
media_image1.png
Greyscale
PNG
media_image2.png
411
408
media_image2.png
Greyscale
Marino further teaches in paragraph [0117], and corresponding Fig. 3C, that “host regions” may be identified in the first frame, which the Examiner interprets as a form of bounding box. Marino then EXPLICITLY teaches of an object being inserted in a first frame fully visible and becoming occluded in a later frame as shown in Figs. 3H and 3K (reproduced below).
PNG
media_image3.png
417
423
media_image3.png
Greyscale
PNG
media_image4.png
406
417
media_image4.png
Greyscale
These teachings of Marino, in particular Fig. 3K, would at least suggest that the object does not in fact “simply stay in the foreground” (further see paragraph [0118]), as Applicant states, but transitions between being occluded or not occluded from one frame to the next.
Double Patenting
The previous rejections of Claims 2-21 on the ground of nonstatutory double patenting are hereby withdrawn due to Applicant’s Terminal Disclaimer filed 3/6/2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-4, 8, and 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luan et al. (US 2018/0122114), hereinafter Luan, in view of Travis et al. (US 2007/0067707), hereinafter Travis, in view of Gaddy et al. (US 2014/0241582), hereinafter Gaddy, in view of Marino et al. (US 2017/0278289), hereinafter Marino.
As per claim 2, Luan teaches the following:
an image processing apparatus configured to detect occlusion of (see abstract), the image processing apparatus comprising a processor configured to:
receive a video. As Luan teaches in paragraph [0036], a video is obtained;
based at least on the computing:
determine that the , (see abstract); and
identify pixels in the second region that are foreground pixels. As Luan teaches in paragraph [0083], each pixel point is analyzed to determine a foreground area and background area of an image;
based at least on identifying the foreground pixels in the second region, determine that a portion of the foreground pixels are occluding at least a portion of the digital ink. As Luan teaches in paragraph [0058], and corresponding Fig. 2, 204, it is determined whether there is an overlap between a business object and a foreground area;
based on determining that the portion of the foreground pixels are occluding at least the portion of the digital ink, hide at least the portion of the digital ink occluded by the portion of the foreground pixels. As Luan teaches in paragraph [0062], the overlapped part of the business object may be excluded; and
update a display of the digital ink in the second frame to reflect the hiding of at least the portion of the digital ink. As Luan teaches in paragraph [0062], the overall image is “drawn”. Further see paragraph [0068] where the modified video is “watched”.
While Luan teaches of inserting a business object into a video (see abstract), Luan does not explicitly teach of the object being digital ink inserted in a first frame of the video. In a similar field of endeavor, Travis teaches of adding content to a video, (see abstract). Travis further teaches the following:
identify an annotation made to a first frame in the video, the annotation being made after the first frame in the video has been captured, the annotation comprising digital ink applied to the first frame in the video. As Travis teaches in paragraph [0043], and corresponding Fig. 5, digital ink annotations may be received corresponding to a media data stream (video). As Travis further teaches in paragraph [0043], the annotations are made while the stream is rendered, thus the video “has already been captured”, and the annotation corresponds to time periods of the video stream. Travis teaches in paragraph [0030], and corresponding Figs. 3A and 3B, that the annotation may be associated with a key frame (first frame).
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the objects of Luan with the digital ink of Travis. One of ordinary skill would have been motivated to have made such modification because as Travis teaches in paragraphs [0001] and [0002], such modification would benefit a user in allowing commenting and annotating using handwritten strokes and combined with videos as a function of time.
Furthermore, Luan in view of Travis does not explicitly teach of a bounding region and comparing said region between frames. In a similar field of endeavor, Gaddy teaches in the abstract of calculating an occlusion area between objects (similar to the overlap function of Luan). Gaddy further teaches the following
for the first frame of the video, compute a model describing pixels in a . Gaddy teaches in paragraph [0012] of receiving a first image of a video frame to compute the first image motion model toward a second image to obtain a motion-compensated image in the occlusion region with pixel values of the marking occlusion with a background image;
for a second frame of the video, identify a second region that corresponds to the . As Gaddy further teaches in paragraph [0012], a second image (second frame) is compared to a first image (first frame) to track movement of an object (bounding region);
compute a comparison between pixels in the second region and the model. As Gaddy teaches in paragraphs [0030] and [0031], pixels are compared between the two images.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the frame comparison of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Furthermore, Luan does not explicitly teach of the ink being not occluded in the first frame and occluded in the second frame and Gaddy does not explicitly teach of a “bounding” region. In a similar field of endeavor, Marino teaches of integrating target content into a source video content (see abstract). Marino further teaches the following:
wherein none of the digital ink is occluded by foreground pixels in the first frame in the video; hide at least the portion of the digital ink occluded by the portion of the foreground pixels in the second frame of the video. As Marino shows in Figs. 3H and 3K that an object inserted into a video may not be occluded in a first frame (3H) and occluded in a second frame some time after the first (3K).
Marino further teaches in paragraph [0119], and corresponding Figs. 3N and 3O, that a rectilinear bounding box may be placed around a marker (digital ink of Gaddy) that defines the placement of an added object and that occlusion is calculated after an object is added to the bounding box.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the occlusion method of Luan with the occlusion differences in first and second frames of Marino. One of ordinary skill would have been motivated to have made such modification because such occlusion determination benefits users in a more realistic appearance of inserted content as being part of the original video. Furthermore, Luan suggests the use of multiple frames of a video in [0077], where foreground objects may change position, and determining whether occlusion does or does not occur in Fig. 2.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the digital ink of Luan in view of Gaddy with the bounding boxes of Marino. One of ordinary skill would have been motivated to have made such modification because as Marino teaches in paragraph [0105], such bounding boxes benefit user in identifying regions where content is added for occlusion and blending purposes.
Regarding claim 3, modified Luan teaches the apparatus of claim 2 as described above. However, as described above, Luan does not explicitly teach of comparing frames. Gaddy further teaches the following:
the comparison comprises a computed similarity value between pixels of the second region and the model. See paragraph [0043], “similarity measures”.
It would have been obvious to one of ordinary skill in the art at the time the application was filed to have further modified the digital ink occlusion of Luan in view of Travis with the similarity measures of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Regarding claim 4, modified Luan teaches the apparatus of claim 3 as described above. However, as described above, Luan does not explicitly teach of comparing frames. Gaddy further teaches the following:
the comparison comprises an occlusion map indicating pixels of the second region which have a similarity value below a predetermined threshold value. As Gaddy teaches in paragraph [0061], an occlusion map is formed with error fields below a given threshold.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the occlusion map of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Regarding claim 8, modified Luan teaches the apparatus of claim 3 as described above. However, as described above, Luan does not explicitly teach of comparing frames. Gaddy further teaches the following:
determine data from the second region. As Gaddy teaches in paragraph [0030] map generator 245 estimates a field of motion between a first and a second image based on data of the second image; and
update the model to describe pixels of the second region which have a similarity value above a predetermined threshold value if the data from the second region has not yet been included in the model. As Gaddy teaches in paragraphs [0034] and [0035], is the map or error field is above a threshold value, the generator repeats comparing and regularizing steps to update the image model.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the model updates of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Regarding claim 12, modified Luan teaches the apparatus of claim 2 as described above. However, as described above, Luan does not explicitly teach of comparing frames. Gaddy further teaches the following:
the bounding region is associated with an object to which the digital ink was applied in the first frame. As Gaddy shows in Fig 1, an image pair is shown with object 115. Upon the modification of Luan in view of Travis in view of Gaddy, the digital ink of Travis may be the object of Gaddy.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the object of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Regarding claim 13, modified Luan teaches the apparatus of claim 2 as described above. However, as described above, Luan does not explicitly teach of comparing frames. Gaddy further teaches the following:
computing the comparison between pixels in the second region and the model further comprises computing a comparison between pixels in the second region and the object to which the digital ink was applied. As Gaddy teaches in paragraph [0056], pixel locations may be filtered with respect to object boundaries. Further see paragraph [0004], where background pixels of an image are occluded behind a foreground object, and would thus require comparison.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the frame comparison of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
A per claim 14, Luan teaches the following:
a computer-implemented method for detecting occlusion of digital ink in a digitally annotated video, (see abstract).
The remaining limitations of claim 14 are substantially similar to those of claim 2 and are rejected using the same reasoning.
Regarding claim 15, modified Luan teaches the method of claim 14 as described above. However, Luan does not explicitly teach of sub-models. Gaddy teaches the following:
the model comprises a set of sub-models describing pixels of the first frame in a set of respective first sub-regions making up the bounding region. As Gaddy teaches in paragraph [0046], the region of support weighting model can be 3x3 or 5x5 of the first image;
the second region comprises a set of second sub-regions corresponding to the set of first sub-regions. As Gaddy teaches in paragraph [0046], the region of support weighting model can be 3x3 or 5x5 of the second image; and
the method further comprises computing a comparison between each pixel of each second sub-region and a corresponding sub-model. As Gaddy teaches in paragraph [0031], the occlusion map generator compares a plurality of pixel values of the motion-compensated second image to a plurality of pixels of the first image with an estimated error field (sub-model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the frame comparison of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Regarding claim 16, modified Luan teaches the method of claim 15 as described above. However, as described above, Luan does not explicitly teach of sub-models. Gaddy teaches the following:
each sub-model describes a cell of a grid of pixels. As Gaddy teaches in paragraph [0046], the region of support weighting model can be 3x3 or 5x5 pixels.
Regarding claim 17, modified Luan teaches themethod of claim 15 as described above. However, Luan does not explicitly teach of a comparison at boundaries between neighboring sub-regions. Gaddy teaches the following:
Further comprising interpolating the comparison at boundaries between neighboring sub-regions. As Gaddy teaches in paragraph [0013], the local weighting over a boundary of the local region of support is based on an eigensystem analysis of the local gradient structure tensor for each pixel (sub-region) of the motion compensated image.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the frame comparison of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
As per claim 18, Luan teaches the following:
one or more computer storage media with device-executable instructions, (see Fig. 8,806).
The remaining limitations of claim 18 are substantially similar to those of claim 15 and are rejected using the same reasoning.
Regarding claims 19 and 20, modified Luan teaches the media of claim 18 as described above. The remaining limitations of claim 19 and 20 are substantially similar to those of claims 15 and 16 respectively, and are rejected using the same reasoning.
Claim(s) 5, 7, 8, 10, 11, and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luan in view of Travis in view of Gaddy in view of Marino as applied to claims 2, 3, and 18-20 above, and further in view of Divakaran et al. (US 2014/0347475), hereinafter Divakaran.
Regarding claim 5, modified Luan teaches the apparatus of claim 3 as described above. However, Luan in view of Travis in view of Gaddy does not explicitly teach an occlusion probability map calculated from similarity of each pixel. In a similar field of endeavor, Divakaran teaches of tracking moving objects in video (see abstract). Divakaran further teaches the following:
the comparison comprises an occlusion probability map calculated from the similarity values of each of the pixels of the second region. As Divakaran teaches in paragraph [0043], a foreground mask (occlusion map) may be formed based on hypothesized regions of the foreground (probability). Divakaran further teaches in paragraph [0030] that a hypotheses may have a degree of certainty, i.e., a similarity value.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the hypothesis mask of Divakaran. One of ordinary skill would have been motivated to have made such modification because as Divakaran teaches in paragraph [0022], such masking method benefits users in more efficient tracking of objects in video in real time.
Regarding claim 7, modified Luan teaches the apparatus of claim 5 as described above. However, Luan in view of Travis in view of Gaddy does not explicitly teach of updating the ink. Divakaran further teaches the following:
update the ink by applying the probability map to the ink. As Divakaran teaches in paragraph [0047] that a bounding box (ink) may be updated with a marking to indicate a degree of confidence (probability map).
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the object updating of Divakaran. One of ordinary skill would have been motivated to have made such modification because as Divakaran teaches in paragraph [0022], such masking method benefits users in more efficient tracking of objects in video in real time.
Regarding claim 8, modified Luan teaches the apparatus of claim 5 as described above. However, Luan in view of Travis in view of Gaddy does not explicitly teach of having a learning rate. Divakaran further teaches the following:
the model is updated according to a learning algorithm having a learning rate. As Divakaran teaches in paragraph [0086], a classifier may learn distributions of same and different objects across different fields of view.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the learning rate of Divakaran. One of ordinary skill would have been motivated to have made such modification because learning systems offer the benefit of improving accuracy over time as the system learns.
Regarding claim 10, modified Luan teaches the apparatus of claim 8 as described above. However, as described above, Luan in view of Travis in view of Gaddy does not explicitly teach of having a learning rate. Divakaran further teaches the following:
the processor is configured to change the learning rate. As Divakaran teaches in paragraph [0086], the system is configured to change the classifier, which would in turn, change the learning rate.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the learning rate of Divakaran. One of ordinary skill would have been motivated to have made such modification because learning systems offer the benefit of improving accuracy over time as the system learns.
Regarding claim 11, modified Luan teaches the apparatus of claim 3 as described above. However, as described above, Luan in view of Travis does not explicitly teach of an occlusion map with similarity values below a threshold value. Gaddy further teaches the following:
compute the comparison by:
generating an occlusion map indicating occluded pixels of the second region which have a similarity value below a predetermined threshold value. As Gaddy teaches in paragraphs [0034] and [0035], and corresponding Fig. 4, if a value based on the occlusion map or error field is above a threshold value, the generator repeats comparing and regularizing steps. Further see paragraph [0043] for similarity values.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the digital ink occlusion of Luan in view of Travis with the occlusion map with similarity values of Gaddy. One of ordinary skill would have been motivated to have made such further modification because as Gaddy teaches in paragraph [0071], such occlusion calculation benefit users with a high-throughput system with improved perceptual quality.
Furthermore, none of Luan, Travis, or Gaddy explicitly teach of marking pixels as occluded depending upon a neighborhood of pixels. Divakaran teaches the following:
for each pixel not indicated as occluded, marking the pixel as occluded if the number of occluded pixels in a selected neighborhood of the pixel is above a predetermined threshold value. As Divakaran teaches in paragraph [0050] and claim 2, the likelihood that a part detection response is missing is large if a body part is occluded by other detection responses with lower Z-values, and thus results in occluded pixels of the second box based on the regularized occlusion map (similarity value) is below a threshold value.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the neighborhood pixels of Divakaran. One of ordinary skill would have been motivated to have made such modification because as Divakaran teaches in paragraph [0022], such masking method benefits users in more efficient tracking of objects in video in real time.
Regarding claim 21, modified Luan teaches the media of claim 20 as described above. The remaining limitations of claim 21 are substantially similar to those of claim 17 and are rejected using the same reasoning.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luan in view of Travis in view of Gaddy in view of Marino in view of Divakaran as applied to claims 2, 3, and 5 above, and further in view of Forsyth et al. (US 2015/0310135), hereinafter Forsyth.
Regarding claim 6, modified Luan teaches the apparatus of claim 5 as described above. However, Luan does not explicitly teach of filtering the probability map. Divakaran teaches the following:
the probability map is filtered. As Divakaran teaches in paragraph [0030], motion constraints are used to filter tracks that are potential candidates for association for a probabilistic likelihood in the occlusion map.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have modified the comparison of Luan in view of Gaddy with the occlusion map filtering of Divakaran. One of ordinary skill would have been motivated to have made such modification because as Divakaran teaches in paragraph [0022], such masking method benefits users in more efficient tracking of objects in video in real time.
Furthermore, none of Travis, Gaddy, nor Divakaran explicitly teach of using a cross bilateral filter. In a similar field of endeavor, Forsyth teaches of a method of processing 3D images (see abstract). Forsyth further teaches the following:
using a cross bilateral filter to generate the comparison. As Forsyth teaches in paragraph [0097], a cross-bilateral filter may be utilized to smooth the result of object identification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of applicant’s claimed invention to have further modified the probability map filtering of Luan in view of Divakaran with the cross-bilateral filter of Forsyth. One of ordinary skill would have been motivated to have made such modification because as Forsyth teaches in paragraph [0005], such techniques benefit users in improved visualization of objects for better design planning.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GREGORY A DISTEFANO whose telephone number is (571)270-1644. The examiner can normally be reached Monday - Friday: 9 am - 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Bashore can be reached at 5712424088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GREGORY A. DISTEFANO/
Examiner
Art Unit 2174
/WILLIAM L BASHORE/ Supervisory Patent Examiner, Art Unit 2174
Prosecution Insights