DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In response to the Non-Final Office Action from 2/2/2026, Applicant has filed an amendment on 5/4/2026. In this reply, Applicant has amended independent 1, 17, and 21 to further specify that reading window information on the display to determine subject matter contained on the display is performed "with Computer Vision (CV)."
Applicant has also argued that the prior art of record fails to teach the limitations added via the instant amendment (Remarks, Pages 7-8). These arguments have been fully considered, however, are moot with respect to the new grounds of rejection, necessitated by the amended claims and further in view of Rochford, et al. (U.S. PG Publication: 2019/0019508 A1).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 23 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 23, Line 5 repeats the term "Computer Vision (CV)" when the term already appears in parent claim 21. It is thus unclear whether the term finds antecedence in the earlier instance of the term or represents a second instance of the term. For claim construction purposes in the interest of compact prosecution, "Computer Vision (CV)" will be construed as --the Computer Vision (CV)--.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 17-26, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Gatzke, et al (U.S. PG Publication: 2024/0211204 A1) in view of Rochford, et al. (U.S. PG Publication: 2019/0019508 A1).
With respect to Claim 1, Gatzke discloses:
A system for operating a program on a primary electronic device comprising:
a primary electronic device including a microphone and a display, and having a memory to store executable instructions and one or more processors, when implementing the executable instructions (processor(s) and memories storing instructions, Paragraph 0015, 0017, and 0051; microphone and video displays, Paragraphs 0014-0015), to:
obtain, with a sensor (see camera, Paragraphs 0021 and 0030), context data related to a user of the primary electronic device (eye gaze tracking that provide a context for a voice command interface operating on an electronic device, Paragraphs 0018, 0020-0021, and 0030);
determine in real time, a first location on the display of the primary electronic device based on the context data (objects on a display are identified based upon gaze tracking (e.g., "UI buttons and/or menu commands" and "to determine when the participant is looking/gazing at a particular voice-enabled UI object"), Paragraphs 0021, 0030, and 0034; the operations for this step and others mentioning "real-time" in Gatzke are performed in the context of streaming conferencing communication (Paragraphs 0013) and utilize high speed networks and computer equipment (Paragraphs 0055-0056), thus it is implied that such operations utilize in conference room-style communications are performed in real-time);
read window information to determine subject matter contained on the display (determination of displayed subject matter located at certain positions such as displayed interactive "UI objects", Paragraphs 0008, 0016, 0018, and 0028; Fig. 2A, windowed Elements associated with 220);
determine in real time, using the microphone of the primary electronic device, a voice command related to the first location on the display (Paragraph 0020- "utter or speak a voice command associated with the given voice-enabled UI object in order to cause the video conference application 120 to perform one or more actions within the video conference;" Paragraph 0030- "media stream generated for the participant 102-1 (e.g., media stream 122-1) can be examined/analyzed to determine whether the participant has spoken a voice command associated with the particular voice-enabled UI object at which the participant is looking/gazing;" See also Paragraphs 0032, 0036, and 0064; device microphone as an audio input, Paragraph 0015); and
actuate a control point in real time at the first location on the display based on the context data, the subject matter contained on the display, and the voice command (Paragraph 0022- "the voice command is associated with or otherwise linked to a particular voice-enabled UI object such that speaking or uttering the voice command by the participant 102-1 may cause performance of one or more actions, operations, etc. within the video conference for the participant (e.g., mute, hang-up, share content, etc.);" Paragraph 0028- "interactive environment 200 may include one or more voice-enabled UI objects 220, each of which can be associated with a particular voice command ( or a particular set of voice commands) that can be uttered or spoken by participant 102-1 in order to cause the video conference application 120 to perform one or more actions within the video conference;” Paragraph 0032- “for the "Share Content" voice-enabled UI object 220-2 at which it is determined that the participant 102-1 is gazing/looking at (as generally shown at 222), if it is further determined that the participant is not an active speaker for the video conference and has spoken the "Share Content" voice command.” See also Paragraphs 0030 and 0035).
While Gatzke is seems to be aware of the icon positions on a display screen in their generation and in that sense reads on the determination contents/subject matter of the display (see Paragraphs 0016 and 0028-0030), Gatzke does not teach that the contents of the screen are read with computer vision in order to determine subject matter. Rochford, however, discloses the use of "computer vision" to identify the object on a display that is the subject matter of eye focus for voice command context (Paragraphs 0067, 0071, 0074, 0077 (discussing particular objects/information in a display window such as an icon or a button as examples), and 0091).
Gatzke and Rochford are analogous art because they are from a similar field of endeavor in the form of multi-modal speech recognition using gaze detection. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Gatzke to detect display objects using the CV-based screen reading taught by Rochford to provide a predictable result in the form of providing additional context about a voice command pertaining to displayed subject matter (Rochford, Paragraph 0074) and keeping up with changing displayed subject matter (Rochford, Paragraph 0091).
With respect to Claim 2, Gatzke further discloses:
The system of claim 1, wherein to determine the voice command the one or more processors are further configured to:
detect a sound in an environment of the user (Paragraph 0031- “monitoring audio obtained for the participant 102-1 via audio I/O device(s)”);
determine the user created the sound (Paragraph 0031- “determine if a volume of audio for the participant 102-1 satisfies (e.g., is greater than or is greater than or equal to) a particular "active speaker" threshold”);
identify at least one word from the sound (Paragraph 0032- “determined that the participant is not an active speaker for the video conference and has spoken the "Share Content" voice command;” note that the particular word(s) of the spoken command are determined;” See also Paragraph 0029 for other command words that may be identified);
convert the at least one word into voice to text data (Paragraph 0032- “determined that the participant is not an active speaker for the video conference and has spoken the "Share Content" voice command;” note that the particular word(s) of the spoken command are determined;” See also Paragraph 0029 for other command words that may be identified; accordingly the system taking in a sound command and converting the command into a particular command word in a system vocabulary is a transcription operation); and
compare the at least one word to a list of words associated with the first location to determine the voice command (different recognized voice commands are matched with different system vocabulary commands (e.g., for muting, hanging up a call, sharing content, etc.) to have those specific commands carried out, Paragraphs 0029 and 0032; for example a user speaking a “share content” command that is recognized is implied to be matched to the share content command in the vocabulary so that this particular function is performed instead of muting or hanging up a call).
With respect to Claim 3, Gatzke further discloses:
The system of claim 1, wherein the one or more processors are further configured to determine a second location on the display in response to actuating the control point at the first location (additional display locations can be identified in response to execution/actuation of the control point such as a display of shared content in response to a “share content” command or in response to a new user gaze location in a UI after a command was issued, Paragraphs 0020, 0029 and 0034).
With respect to Claim 4, Gatzke further discloses:
The system of claim 1, wherein in the one or more processors are further configured to: determine whether data is stored within the memory that defines the control point; and dynamically adjust the control point, in real time, in the memory in response to determining the control point is in the memory ("gaze tracking logic" that keeps track of "eye tracking/gaze details" to determine whether a user is looking towards "one or more voice-enabled user interface objects," Paragraph 0015, 0021, 0025-0026, and 0045; data and information is tracked/stored in memory, Paragraph 0054; tracking information is ongoing and dynamic and can identify status when and object is being looked at and when gaze ceases, Paragraph 0034).
With respect to Claim 5, Gatzke further discloses:
The system of claim 1, wherein the primary electronic device includes at least one sensor in communication with the one or more processors, and the one or more processors obtain the context data from the at least one sensor (Paragraph 0030- “eye tracking/gaze details/information generated via video camera(s) 112;” See also Paragraph 0021).
With respect to Claim 6, Gatzke further discloses:
The system of claim 1, wherein the one or more processors utilize computer vison to determine the first location on the display (computer-based eye/gaze tracking, Paragraphs 0015, 0018, and 0021).
With respect to Claim 7, Gatzke further discloses:
The system of claim 1, wherein the one or more processors are further configured to obtain context data from a communication from a secondary electronic device (secondary electronic device such as a video headset obtains context data in the form of gaze, Paragraphs 0012, 0017, and 0021; Fig. 1, Elements 110-1 and 130-1 showing a multi-device environment; see also the use of an audio I/O device for obtaining audio-based context information to determine an active speaker, Paragraph 0031).
With respect to Claim 8, Gatzke further discloses:
The system of claim 1, the one or more processors are further configured to determine the control point is within the first location before actuating the control point (detection of a user looking at a particular location/UI object within a display prior to voice command actuation/operation (e.g., muting a call or sharing content), Paragraphs 0020, 0026, and 0030).
Claim 17 relates to the method practiced by the system of claim 1 embodied as a non-transitory computer readable storage medium comprising computer executable code, and thus, contains similar subject matter. Accordingly, claim 17 is rejected for reasons similar to claim 1. Furthermore, Gatzke discloses method embodiment as a non-transitory computer readable storage medium comprising computer executable code (Paragraphs 0062-0063).
Claims 18-20 contain subject matter respectively similar to claims 2-4, and thus, are rejected under similar rationale.
With respect to Claim 21, Gatzke disclsoes:
A system for operating a program on a primary electronic device comprising: a primary electronic device having a memory to store executable instructions and one or more processors, when implementing the executable instructions (processor(s) and memories storing instructions, Paragraph 0015, 0017, and 0051; multi-device operating environment, Fig. 1, e.g., Elements 110-1 and 130-1), to:
obtain, with a sensor (see camera, Paragraphs 0021 and 0030), context data related to a user of the primary electronic device (eye gaze tracking that provide a context for a voice command interface operating on an electronic device, Paragraphs 0018, 0020-0021, and 0030);
determine subject matter contained on the display (determination of displayed subject matter such as displayed interactive "UI objects", Paragraphs 0008, 0018, and 0028; Fig. 2A, windowed Elements associated with 220);
determine whether the program is muting the user based on the context data (context data in the form of gaze used to determine that a user is muted, Paragraph 0026, 0029, and 0034);
read window information on the display (determination of displayed subject matter such as displayed interactive "UI objects", Paragraphs 0008, 0018, and 0028; Fig. 2A, windowed Elements associated with 220) to determine, based on the context data and the subject matter contained on the display in real time, whether the user intends to communicate using sound via the program in response to determining the program is muting the user (again relying on gaze information of UI objects (“no longer looking at”) to determine that a user intends to speak, Paragraphs 0030, 0034, 0049, and 0065; the operations for this step and others mentioning "real-time" in Gatzke are performed in the context of streaming conferencing communication (Paragraphs 0013) and utilize high speed networks and computer equipment (Paragraphs 0055-0056), thus it is implied that such operations utilize in conference room-style communications are performed in real-time); and
automatically actuate the program to unmute a microphone in real time to allow the sound of the user to be communicated to the program when determining that user intends to communicate using the sound (unmuting operation is automatically performed based on the context information in the form of gaze tracking, Paragraphs 0034, 0049, and 0065; audio input is received via a microphone that is muted if mute is active, Paragraph 0015).
While Gatzke is seems to be aware of the icon positions on a display screen in their generation and in that sense reads on the determination contents/subject matter of the display (see Paragraphs 0016 and 0028-0030), Gatzke does not teach that the contents of the screen are read with computer vision in order to determine subject matter. Rochford, however, discloses the use of "computer vision" to identify the object on a display that is the subject matter of eye focus for voice command context (Paragraphs 0067, 0071, 0074, 0077 (discussing particular objects/information in a display window such as an icon or a button as examples), and 0091).
Gatzke and Rochford are analogous art because they are from a similar field of endeavor in the form of multi-modal speech recognition using gaze detection. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Gatzke to detect display objects using the CV-based screen reading taught by Rochford to provide a predictable result in the form of providing additional context about a voice command pertaining to displayed subject matter (Rochford, Paragraph 0074) and keeping up with changing displayed subject matter (Rochford, Paragraph 0091).
With respect to Claim 22, Gatzke disclsoes:
The system of claim 21, wherein the program is a conference calling application (note that claim 22 further limits an intended use recitation (i.e., “for operating a program) that does not structurally limit the system of claim 21, however, it is nevertheless worth noting that Gatzke discloses a conference calling application in the form of a “video conference application” such as Webex, Paragraph 0011).
With respect to Claim 23, Gatzke further discloses:
The system of claim 21, wherein to determine the user intends to communicate using sound the one or more processors are configured to: analyze the context data to determine the user is in front of the primary electronic device (the gaze tracking is capable of detecting a user looking directly at the primary device wherein “directly at” corresponds to a user being in front of the device, Paragraphs 0026); and
determine, using Computer Vision (CV), a gaze of the user is looking at a working space ((computer-based eye/gaze tracking is used on the part of human tracking, Paragraphs 0015, 0018, and 0021, pertaining to looking at a computer-based conference work space, Paragraph 0016 and Fig. 3A element 300).
With respect to Claim 24, Gatzke further discloses:
The system of claim 21, further comprising: the sensor coupled to the one or more processors and configured to obtain the context data (Paragraph 0030- “eye tracking/gaze details/information generated via video camera(s) 112;” See also Paragraph 0021; see Fig. 5 showing a communication bus (508) coupling I/O sensors to the processor(s)).
With respect to Claim 25, Gatzke further discloses:
The system of claim 24, wherein the sensor is at least one of a camera, microphone, infrared sensor, or temperature sensor (Paragraph 0030- “eye tracking/gaze details/information generated via video camera(s) 112).
With respect to Claim 26, Gatzke further discloses:
The system of claim 21, wherein the one or more processors are further configured to: determine a working space of the user (physical work environment recognition, Paragraph 0016); and determine whether the user is within the working space in front of the primary electronic device (the gaze tracking is capable of detecting a user looking directly at the primary device wherein “directly at” corresponds to a user being in front of the device, Paragraphs 0015, 0018, 0021, and 0026).
With respect to Claim 28, Gatzke further discloses:
The system of claim 21, wherein the one or more processors are further configured to: determine the sound of the user is a voice command (Paragraph 0032- “determined that the participant is not an active speaker for the video conference and has spoken the "Share Content" voice command;” note that the particular word(s) of the spoken command are determined;” See also Paragraph 0029 for other command words that may be identified); and
implement the voice command (Paragraph 0022- "the voice command is associated with or otherwise linked to a particular voice-enabled UI object such that speaking or uttering the voice command by the participant 102-1 may cause performance of one or more actions, operations, etc. within the video conference for the participant (e.g., mute, hang-up, share content, etc.)).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Gatzke, et al. in view of Rochford, et al. and further in view of Terrano (U.S. PG Publication: 2020/0090401 A1).
With respect to Claim 9, Gatzke Rochford teaches the gaze-directed voice command execution system as applied to claim 1. Although Gatzke teaches that gaze tracking may employ any “techniques/logic/algorithms now known in the art” (Paragraph 0021), Gatzke in view of Rochford does not specifically teach utilizing an artificial intelligence application to analyze the context data. Terrano, however, discloses using artificial intelligence (AI) software (see “machine learning”) to analyze context data in the form of gaze tracking information (Paragraphs 0021 and 0038).
Gatzke, Rochford and Terrano are analogous art because they are from a similar field of endeavor in voice command processing using gaze detection. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to use the gaze tracking using artificial intelligence as taught by Terrano as one of the techniques used to analyze gaze taught by Gatzke in view of Rochford to provide a predictable result of using an algorithm that can learn how to effectively track a gaze direction of a user.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Gatzke, et al. in view of Rochford, et al. and further in view of Jorasch, et al. (U.S. PG Publication: 2021/0399911 A1).
With respect to Claim 27, Gatzke in view of Rochford teaches the system for video conference management using context in the form of gaze tracking and muting functionality as applied to Claim 26. Although Gatzke teaches the removal of “another individual” (e.g., family) or animal (e.g., “dog”) from conference audio (Paragraph 0023), Gatzke in view of Rochford does not describe their identification and removal from image data. Jorasch, however, disclsoes the identification of other objects in a video conference such as other people or children and their removal from the scene (Paragraphs 2514 and 2834).
Gatzke, Rochford, and Jorasch are analogous are because they are from a similar field of endeavor in voice communication interfaces. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to use the image filtering taught by Jorasch in the video conference application taught by Gatzke in view of Rochford in order to provide a predictable result of removing distractions in a video conference.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Blume, et al. (U.S. PG Publication: 2019/0250873 A1)- teaches the use of computer vision techniques to detect content on a display (Paragraphs 0033 and 0037).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES S WOZNIAK whose telephone number is (571)272-7632. The examiner can normally be reached 7-3, off alternate Fridays.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant may use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAMES S. WOZNIAK
Primary Examiner
Art Unit 2655
/JAMES S WOZNIAK/Primary Examiner, Art Unit 2655
Prosecution Insights