Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 21 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, regards as the invention.
Regarding claim 21, this claim recites that the plurality of overlays corresponds to a plurality of letters to enable a letter-by-letter display “and stop” when a portion of any letter is covered. However, the grammar and terminology of the phrase “and stop” makes it unclear as to what is being stopped and how that relates to the letter-by-letter display. For example, is the letter-by-letter display stopped? Is display of the overlay stopped? And so on. For the purposes of examination, and to be consistent with the specification, this will be interpreted to mean that display of the letters can be stopped.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10, 12-19, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over McHugh et al. (U.S. Patent Application Publication No. 2019/0279407), referred herein as McHugh, in view of Storer (U.S. Patent Application Publication No. 2020/0314356), referred herein as Storer.
Regarding claim 1, McHugh teaches a method for controlling a display of an overlay in a video of a scene (figs 4, 6, 8, and 9; paragraph 94, lines 1-7; paragraph 95, lines 1-2), comprising: defining the overlay to be displayed or stopped in the video, defining an overlay area including the overlay (figs 6A, B, and D, overlay 610, overlay area 605; paragraph 96, lines 1-8); defining at least one foreground object of the scene in the video (figs 6A, B, and D, foreground object 615; paragraph 96, the last 4 lines); determining whether a spatial overlap is present in a video frame of the video between the foreground object and the overlay area, and in response to determining the spatial overlap in the video frame, stopping displaying a display of the overlay within the overlay area (fig 6D; paragraph 97, lines 1-5 and the last 14 lines; paragraph 108, lines 1-12; paragraph 109, lines 1-11; paragraphs 145 and 148; when foreground object 615 is determined to spatially overlap with overlay 610 in overlay area 605, the display of the entire overlay 610 is stopped, e.g. by fading out the overlay using a change in transparency/alpha [which is consistent with the application’s specification and dependent claims 3 and 4, for example]).
McHugh teaches fading thresholds (paragraphs 132 and 182) and modifying pixels values based on threshold distances (paragraph 163), but does not explicitly teach determining a size of the spatial overlap and whether the size exceeds a first threshold, wherein the display of the overlay is changed in response to determining that the size of the spatial overlap exceeds the first threshold.
However, in a similar field of endeavor, Storer teaches a method for controlling the display of an overlay in a video of a scene, comprising determining a foreground object of the scene and a background, and adjusting alpha values accordingly (paragraph 12; paragraph 63; paragraphs 74 and 79), and further comprising determining a size of a spatial overlap and whether the size exceeds a first threshold, wherein a display of the overlay is changed in response to determining that the size of the spatial overlap exceeds the first threshold (paragraph 64; paragraph 74, lines 1-4 and the last 3 lines; paragraph 80, the last 18 lines; paragraph 81, lines 1-11; the overlap size can be determined by the number of pixels, a percentage amount of overlay, etc., and has a determined threshold; foreground objects are overlayed on the background in various ways that are based on the size determination).
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to combine the spatial overlap threshold of Storer with the image processing of McHugh because this provides a faster and more efficient way to evaluate the overlap and process the image accordingly, thereby increasing the efficiency of the processing while reducing bandwidth and storage requirements (see, for example, Storer, paragraph 14).
Regarding claim 2, McHugh in view of Storer teaches the method of claim 1, wherein the size of the first threshold is related to a size of the overlay area (Storer, paragraph 64; paragraph 80, the last 18 lines; paragraph 81, lines 1-11; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 3, McHugh in view of Storer teaches the method of claim 1, wherein stopping the display of the overlay includes changing an alpha value of pixels included in the overlay area (McHugh, paragraph 131, the last 6 lines; paragraph 132; paragraph 145; Storer, paragraph 79; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 4, McHugh in view of Storer teaches the method of claim 3, wherein the alpha value is changed at a pre-determined rate to achieve a fade-out effect (McHugh, paragraph 131, the last 6 lines; paragraph 132; paragraph 145; Storer, paragraph 79; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 5, McHugh in view of Storer teaches the method of claim 4, wherein the pre-determined rate is selectable by a user (McHugh, paragraphs 100 and 101; paragraph 132; paragraph 135, lines 16-31; Storer, paragraph 79; paragraph 80, the last 7 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 6, McHugh in view of Storer teaches the method of claim 1, further comprising: reintroducing the overlay when the size of the spatial overlap falls below a second threshold (McHugh, paragraph 93, the last 6 lines; paragraphs 112 and 145; Storer, paragraph 64; paragraph 80, the last 18 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 7, McHugh in view of Storer teaches the method of claim 6, wherein the second threshold is the same as the first threshold (McHugh, paragraph 93, the last 6 lines; paragraphs 112 and 145; Storer, paragraph 64; paragraph 80, the last 18 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 8, McHugh in view of Storer teaches the method of claim 6, further comprising starting a timer in response to the stopping of displaying the entire overlay within the overlay area, and wherein reintroducing the overlay occurs when the timer reaches a predetermined value (McHugh, paragraph 108, the last 16 lines; paragraphs 112 and 145; Storer, paragraph 62, lines 1-7; paragraph 63, the last 11 lines; paragraph 80, lines 5-20; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 9, McHugh in view of Storer teaches the method of claim 6, wherein reintroducing the overlay includes changing an alpha value of pixels included in the overlay (McHugh, paragraph 112; paragraph 131, the last 6 lines; paragraphs 132 and 145; Storer, paragraph 79; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 10, McHugh in view of Storer teaches the method of claim 9, wherein the alpha value is changed at a pre-determined rate to achieve a fade-in effect (McHugh, paragraph 91; paragraph 93, the last 6 lines; paragraph 97, the last 14 lines; paragraph 132; Storer, paragraph 79; paragraph 80, the last 12 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 12, McHugh in view of Storer teaches the method of claim 1, wherein determining the spatial overlap is done by determining a ratio between a number of pixels in the overlay area that belong to the foreground object and a total number of pixels in the overlay area (Storer, paragraph 64; paragraph 80, the last 18 lines; the percentage is a ratio; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 13, McHugh in view of Storer teaches the method of claim 12, wherein determining the spatial overlap further takes into account a position within the overlay area of the pixels belonging to the foreground object (McHugh, paragraph 163; Storer, paragraph 35, lines 1-4 and the last 9 lines; paragraphs 37 and 79; paragraph 80, the last 18 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 14, McHugh in view of Storer teaches the method of any claim 1, wherein defining at least one foreground object in the video is done by using a motion-based foreground-background detection algorithm (Storer, paragraphs 35 & 72; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 15, McHugh in view of Storer teaches the method of claim 1, wherein defining at least one foreground object in the video is done by using an object detection algorithm, and classifying pixels belonging to detected objects as foreground pixels and pixels not belonging to detected objects as background pixels (McHugh, paragraphs 71 and 100; Storer, paragraphs 37 and 72; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 16, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the medium, instructions, and computer, which are disclosed by McHugh, fig 2, medium 260 and its stored instructions, processor 240, and paragraphs 53 and 55); thus they are rejected on similar grounds.
Regarding claim 17, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the processor, memory, and instructions, which are disclosed by McHugh, fig 2, processor 240, memory 260 and its stored instructions, and paragraphs 53 and 55); thus they are rejected on similar grounds.
Regarding claim 18, McHugh in view of Storer teaches the method of claim 1, wherein the first threshold comprises a percentage of pixels within the overlay area being covered by the foreground object (McHugh, paragraphs 150 and 152; Storer, paragraph 64, the last 3 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 19, McHugh in view of Storer teaches the method of claim 1, wherein the method comprises automatically setting the first threshold based on an appearance of the scene (Storer, paragraphs 64, 78, and 80; the threshold may be set based on positions, overlap, bounding boxes, etc. of objects in the scene; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Regarding claim 22, McHugh in view of Storer teaches the method of claim 1, in response to determining that the size of the spatial overlap in the video frame does not exceed the first threshold, displaying the entire overlay within the overlay area (McHugh, paragraph 93, the last 6 lines; paragraphs 112 and 145; Storer, paragraph 64; paragraph 80, the last 18 lines; the motivation to combine is similar to that discussed above in the rejection of claim 1).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over McHugh, in view of Storer, and further in view of Yen (U.S. Patent Application Publication No. 2024/0177479), referred herein as Yen.
Regarding claim 21, McHugh in view of Storer teaches the method of claim 1, wherein the overlay is one of a plurality of overlays comprising a plurality of letters to enable an overlay display and stop when a certain portion of the overlay is covered by the foreground object (McHugh, figs 5, 8C, and 8D; the letters in objects 515a, 515b, 808a, etc.; paragraph 91, lines 1-4 and the last 4 lines’ paragraph 92, lines 1-7; paragraph 93; paragraph 129; paragraph 130, the last 7 lines).
McHugh in view of Storer does not teach a plurality of overlays corresponding to a plurality of letters to enable a letter-by-letter display and stop when a portion of any letter is covered by the foreground object.
However, in a similar field of endeavor, Yen teaches a method for controlling a display of an overlay in a scene, comprising defining the overlay and at least one foreground object of the scene, wherein the overlay is one of a plurality of overlays corresponding to a plurality of letters to enable a letter-by-letter display (figs 3A-3C and 3G; paragraph 31; paragraph 33, the last 8 lines; paragraph 48, the last 7 lines; the 3D letter representations overlaying the letter cards in the scene, and the car foreground object; each 3D overlay is one of a plurality of 3D overlays corresponding to a plurality of letters, and a letter-by-letter display is enabled, shown in figs 3A-3C), and stop when a portion of any letter is covered by a foreground object (figs 3F-3I; paragraph 49, lines 1-7; when the determination result is positive, the foreground object emerges and covers a portion of one or more letters, and display of the 3D letter overlays then stops).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the letter display of Yen with that of McHugh in view of Storer because this allows object data to be processed on an individual object basis, thereby saving computational resources and avoiding a negative impact of the performance of the displaying (see, for example, Yen, paragraph 50).
Response to Arguments
Applicant’s arguments with respect to the 112 rejection have been fully considered and are persuasive. The amendments have overcome this rejection; thus it is withdrawn.
On pages 7 and 8 of the Applicant’s Remarks, with respect to the 103 rejection of claim 1, the Applicant again argues that 1) the virtual object of McHugh cannot be regarded as a foreground object because the virtual object is in focus as decided by user input or eye tracking, and 2) the pinned element is a virtual object, not an object in the scene that discloses at least one foreground object. The Examiner respectfully disagrees with these arguments.
Regarding the first argument, it is first noted that McHugh is replete with explicit disclosure of foreground objects – for example, figures 4-6, 8, and 9 all illustrate foreground objects (even ignoring the explicit disclosure in the description), and paragraphs 71, 97, 100, and others explicitly disclose foreground objects. Indeed, the Applicant cites figures 6A-6D, which explicitly disclose foreground objects on an overlay area in several ways. For example, in figure 6A the object 610a may be considered a foreground object that is in front of an overlay area depicted as the shaded region; in figure 6B, objects 610a and 615a are foreground objects in front of an overlay area depicted as the shaded region; in figure 6C, object 610a is a foreground object in front of object 615b, which in turn is in front of an overlay area depicted as the shaded region; and in figure 6D, object 615c is a foreground object in front of object 610b, which in turn is in front of an overlay area depicted as the shaded region; and so on. Further, as is very well-known in the art, a foreground object is simply something in front of a background – thus even absent such explicit disclosure in McHugh, one would still readily recognize the objects in McHugh as foreground objects. Whether objects are brought into focus by user input or eye tracking is irrelevant to whether the objects are considered foreground objects.
Similarly, and in regard to the second argument, the objects in McHugh (and Storer) are very clearly foreground objects, and whether or not they are virtual is irrelevant to whether the objects are considered to be foreground objects. (And this is aside from the fact that figures 8A-8F of McHugh explicitly show an AR environment comprising both real and virtual foreground and background objects and overlay areas.) Indeed, Applicant’s disclosed objects are virtual, yet this does not affect whether they are considered foreground objects.
Any potential distinction between a “foreground object” in the claims and a “foreground object” in the prior art is not currently reflected in the claims.
On pages 8 and 9 of the Applicant’s Remarks, with respect to the 103 rejection of claim 1, the Applicant again argues that Storer detects moving objects by comparing an overlap of the same feature appearing in different frames, thus there is no comparison of a moving object to an overlay. The Examiner respectfully disagrees with this argument.
It is first noted that McHugh clearly teaches determining spatial overlap and comparing foreground object movement to the overlay, as discussed above; thus Storer is not relied upon to teach this feature. Nevertheless, it is respectfully submitted that the argument incorrectly conflates the temporal and spatial analysis of Storer. As described in Storer, and as is well-known in the art, the temporal frame by frame analysis detects movement of an object in the video. But the overlap of an object is a spatial comparison within any one frame, regardless of whether the object is being compared to another object, a background or overlay, or itself in a previous frame. The amount of overlap can change over time, but the state of having overlap is necessarily a spatial determination, and begins at a point in time (a frame) when spatial overlap occurs. This same concept is shown in Applicant’s figure 2B, for example, which shows video frames 218, 220, and 222 at times T1, T2, and T3, respectively. Motion occurs from frame to frame, as the car moves across the image – but only at frame 220 is an overlap illustrated as occurring. Applying Applicant’s argument to this figure, it can be seen that the disclosed invention itself is “comparing an overlap of the same feature appearing in different frames,” e.g. comparing an overlap of the same car appearing in different frames. It is true that in Applicant’s figure 2B, the overlap of interest is a frame by frame comparison of the car 212 to the object 216, while in Storer, the overlap is determined by doing a frame by frame comparison of a bounding box in one frame to a bounding box in a previous frame. But as discussed above, Storer is not relied upon to teach comparison of a foreground object to an overlay area, as this is explicitly taught by McHugh, and Storer’s teaching of determining a size of a spatial overlap compared to a threshold is very clear, and was properly combined with McHugh. Thus, this does not present any distinction between the prior art and the claims.
On page 9 of the Applicant’s Remarks, the Applicant argues that independent claims 16 and 17 are not taught by the prior art for reasons similar to those discussed in regard to claim 1, and that the remaining dependent claims are not taught, insomuch as they depend from claims that are not taught. The Examiner respectfully disagrees with this argument, for the reasons discussed above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613