Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed June 26 2026 has been entered and considered. Claims 1-4, 8-11, and 16 have been amended. In light of the amendment the prior art rejections of claims 1, 8, and 16 are withdrawn as moot. The new grounds of rejection set forth in the present action were necessitated by Applicants’ claim amendments; accordingly, this action is made final.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 16, “identify a display area in the ROI based on a change in luminance in consecutive frames of the continuous image frames, and edges of the ROI are determined by the change in the luminance in the consecutive frames,” is recited in lines 9-11 and “determine metadata from the edges of the determined display area” is recited in line 13. It is not clear whether “edges of the ROI” and “the edges of the determined display area” refer to the same boundary or two different ones. The “display area” is described as being inside the “ROI”, however there is no difference specified between the edges of the “ROI” and the edges of the “display area”, leaving it unclear which edges the metadata is being determined from.
Claims 17-20 inherit the deficiencies of claim 16 and are similarly rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 16-17 and 20 as best understood are rejected under 35 U.S.C. 103 as being unpatentable over Hoarty et al. (Previously cited) in view of Lee et al. (US Patent Pub. No. 2018/0005387 A1, published 2018).
Regarding claim 16, Hoarty teaches an electronic device, for visual content communication, the electronic device comprising: a camera configured to capture a set of continuous image frames displayed on an external device (Para. 120, “The computing device can include any suitable device, such as a display device (e.g., a television), a camera”); a memory storing instructions (Para. 124, “couples various computing device components including computing device memory”); and at least one processor configured to execute the instructions to: identify a region of interest (ROI) from the continuous image frames (Para. 113, “At block 1504, the process 1500 includes identifying a region of pixels of the video frame”), obtain the ROI data in an RGB color space (Para. 62, “On a color display, all signals are used, and the original RGB information is decoded.”), identify a display area in the ROI (Para. 113 above, a ROI is detected, but no specific method is described for this detection), convert data in the display area from the RGB color space to a YCbCr color space (Para. 69, “With this understanding, the techniques described herein will be described in terms of HSL which is translated to RGB and then translated to Y′CbCr”), determine metadata from edges of the determined display area, each of the edges of the determined display area having a fixed pixel width (Para. 100, “An optional (as indicated by the dashed lines) pseudo-random encode area shift engine 1105 can determine a position or location of a pixel region (e.g., one of the positions shown in”; Also see Figs. 5A-5P where the position information at the edges of the determined display area is disclosed, see analysis under “Response to Arguments”), based on a predefined modification factor of a modulation of a Cb component (Para. 115, “In some examples, the at least pixel characteristic of the subset of pixels can include at least one of a hue, a saturation, or a lightness of the region of pixels.”; Para. 102, “The local pixel array analysis engine 1106 can analyze the surrounding pixels for each pixel region of the video frame. Once the subset of pixels (the data carrying block) in a pixel region is determined and the surrounding pixel area around the subset of pixels is examined to determine the pixel characteristics of the surrounding pixels, pixel shift engine 1107 can determine the shift in the pixel data that will be used to encode the data value, either algorithmically or via a LUT or a combination of both, as described above.”), identify a location of a content block from the metadata (Para. 93, “As shown in FIG. 4A, a data carrying block 410 is in a first location in the pixel region of a first frame 401, and is relocated to a second location in the pixel region of a second frame 402 shown in FIG. 4B.”), determine a modification factor based on a change in the Cb component of the content block (Para. 81, “The H, S, and/or L shifts used to alter the data carrying pixels (e.g., the pixels in the core area, such as pixel 205, and/or the pixels in the intermediate area, such as pixel 203) can be determined to some degree algorithmically. For instance, the surrounding reference area (the white pixels) can be sampled to determine the H, S, and/or L (or other pixel characteristic) of the surrounding pixels. Based on the determined H, S, and/or L values (or other pixel characteristic), the pixels in the core area and the pixels in the intermediate area can be can modulated or altered based.”), and obtain an auxiliary content based on binary bits of the modification factor (Para. 96, “In the case of a data carrying block location determined by the watermark data…”; Para. 99, “The encoding system can then receive or obtain watermark data 1102 for a frame. In some cases, a state counter 1103 is initiated to track pseudo-random sequence placement of certain encoded pixels. For example, the state counter 1103 can store pixel modification decisions from the pixel shift engine 1107 (and in some cases from the local pixel array analysis engine 1106 and/or the pseudo random encode area shift engine 1105).”).
Hoarty does not explicitly disclose identifying a display area in the ROI based on a change in luminance in consecutive frames. However, they do determine a position or location of a pixel region using an unspecified method (Para. 100).
Lee teaches to identify a display area in the ROI based on a change in luminance in consecutive frames of the continuous image frames (Para. 33-35, “Next, at operation 620, one or more static regions are detected in the current video frame. The detection is based on differences between selected features extracted from the current video frame and corresponding selected features extracted from a previous video frame. If the differences are less than a threshold value, the region is determined to be static (e.g., relatively unchanged from one frame to the next). The selected features for static region detection include a count of the number of HGR starts and the number of VGR starts for each of the rows of pixels and for each of the columns of pixels. The selected features also include a summation of locations of starting points of the HGRs for each of the rows and a summation of locations of starting points of the VGRs for each of the columns.
At operation 630, boundaries of the static regions are detected based on a location of a maximum value of the count of HGR starts and VGR starts within the static region, if the maximum value exceeds a boundary detection threshold value.”, the “selected features” are luminance values), and edges of the ROI are determined by the change in the luminance in the consecutive frames (Para. 35, “At operation 640, an active region in the current video frame is defined based on exclusion of the detected static regions. The boundaries of the static regions and the active region may be defined by coordinates identifying top left, bottom left, top right, and bottom right pixels of these regions.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hoarty to incorporate the teachings of Lee to include edges of the ROI are determined by the change in the luminance in the consecutive frames. Hoarty teaches a system where a display area is detected as a region of interest and those pixels are used in further processing, however no specific method is provided to detect the region. Lee provides a method where a video can be analyzed to determine an active region with edges, and a change in luminance between frames is used for this detection. One of ordinary skill in the art would have recognized the advantage of implementing the method of Lee into the system of Hoarty, given that the region being detected is a display screen which is predisposed to having changes in luminance across different frames.
Regarding claim 17, Hoarty as modified above teaches all of the elements of claim 16, as stated above, as well as wherein the at least one processor is further configured to execute the instructions to: extract the Cb component and a Cr component of pixels from the ROI (Para. 81, “In some cases, the one or more pixel characteristics can include hue (H), saturation (S), and lightness (L).”, The pixel characteristics are not limited, Cb and Cr are an interchangeable pixel characteristic, as described above); obtain the predefined modification factor (Para. 84, “In some examples, a pixel-value look-up table (LUT) might be used where HSL values are used to look up in a key/value table a recommended substitute value of HSL for the data carrying pixels (in the core area and/or in the intermediate area).”); determine a change in the Cb component and the Cr component in successive frames of the ROI based on the modification factor (Para. 88, “read the value for each HSL parameter from the table entry to be applied to the subset of pixels in the pixel region 102 (e.g., to all pixels in the subset of pixels, as shown in FIG. 1, or to the core area and the intermediate area, as shown in FIG. 2) in percent of full-scale of each respective H, S, and L parameter”); and extract the metadata from the edges of the determined display area based on the change in the Cb component and the Cr component in successive frames of the ROI (Para. 94, “FIG. 5A-FIG. 5P are diagrams illustrating an example of a data carrying block 510 being moved to a multiplicity of locations within the pixel patch area in order to encode multiple bits per pixel patch. As shown, the data encoded pixels of the data carrying block 510 are moved to, in this example, one of 16 positions within the pixel patch area, thus encoding four bits of information.”, Data carrying blocks hold the modified pixels in order to encode additional bits of information. As seen in Figs. 5A-FP, they are at the edges of the display).
Regarding claim 20, Hoarty as modified above teaches all of the elements of claim 16, as stated above, as well as to identify luminance of all the pixels between successive frames; based on the luminance and a luminance factor, detect coordinates of the ROI in successive frames (Lee; Para. 35, “At operation 640, an active region in the current video frame is defined based on exclusion of the detected static regions. The boundaries of the static regions and the active region may be defined by coordinates identifying top left, bottom left, top right, and bottom right pixels of these regions.”).
Claim(s) 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Hoarty et al. as modified in view of Lee et al. above with respect to Claim 16, further in view of Lundbæk (Previously cited).
Regarding claim 18, Hoarty as modified in view of Lee teaches all of the elements of claim 16, as stated above, as well as filtering the modified second set of pixels to obtain user specific data (Para. 111, “The watermark data can act as a trigger for a process in a smart TV to substitute alternative content (e.g., a video, an alternative ad, content from the Internet, or other alternative content) based on demographic and/or user data provided to the TV”, The encoded data is filtered based on user data); select at least one format and at least one medium for displaying the user specific data (Para. 111 above, showcases a format and medium for display); and display the user specific data on the at least one medium (Para. 111 above, showcases a medium for displaying user specific data).
They do not explicitly disclose generating a user association map based on a plurality of interactions of a user with a plurality of entities. However, they do take user data/demographics into account when determining what additional information or content is to be displayed.
Lundbæk teaches to generate a user association map based on a plurality of interactions of a user with a plurality of entities (Para. 170, “A center of interest embodies context data representing a location in the context mapping space 600 that corresponds to a context a user profile is likely to be interested in. The center of disinterest may be determined based on user-specific interest data, including without limitation previously-selected search results, biographical information, previous search queries, and/or the like.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hoarty in view of Lee to incorporate the teachings of Lundbæk to include the generation of a user association map based on a plurality of interactions of a user with a plurality of entities. The method of Hoarty discloses providing alternative content or information to a user based on data that has been previously determined. One of ordinary skill in the art would recognize the advantage of creating a user association map for this purpose, as disclosed by Lundbæk, to allow the system to provide more specific content that aligns with a user’s preferences.
Regarding claim 19, Hoarty as modified above teaches all of the elements of claim 18, as stated above, as well as teaching wherein the at least one medium is an Internet of Things (IoT) device that is selected using an IoT state map (Hoarty, Para. 69, “In some cases, the receiving device can receive and process the video data (e.g., a set-top box, a server, or the like), and can provide the processed video data to another receiving device (e.g., a mobile device, smart television, or the like)”, Smart TV is understood as an IoT device).
Allowable Subject Matter
Claim 1-15 are allowed.
Conclusion
Pertinent Prior art: Shi et al., Chinese Patent Pub. No. 113222805 A, “Fast And High-precision Visual Processing Method For NAO Type Football Robot”, published 8/6/21, Machine translation provided Similar YCbCr processing method which includes a threshold range. Does not select a second set of pixels having a maximum inter-pixel distance and a specific luminance value.
Huang et al., Chinese Patent Pub. No. 111611940 A, “A Fast Video Face Recognition Method Based On Large Data Processing”, published 9/1/2020, Machine translation provided Similar YCbCr processing method but towards facial recognition, includes a modulation threshold range. Does not select a second set of pixels having a maximum inter-pixel distance and a specific luminance value.
Kim Jong Ok et al., Korean Patent Pub. No. 102123166 B1, “A Color Modulation Based Visual Light Communication Method And Apparatus”, published 6/26/2020, Machine translation provided Similar color modulation method working with visible light communication to hide data in an original frame. Does not select a second set of pixels having a maximum inter-pixel distance and a specific luminance value.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights