DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
1. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
2. Claim(s) 1, 6, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360).
Regarding claim 1, Whyte teaches a system comprising: a visual sensor located in a room; an acoustic sensor located in the room; a processing unit connected to the visual sensor and the acoustic sensor (see fig. 1, ¶ 0014-0015, 0018. The system having a microphone array that receives audio data from the room (14) , a camera that includes a color camera, such as an RGB camera, that captures color image data from the room (14).); and a calibration module of the processing unit, the calibration module comprising circuitry configured to translate data accumulated from the visual sensor into spatial data that corresponds with a physical location of the acoustic sensor in the room and a field of view operating parameter for the visual sensor (see fig. 1, ¶ 0017-0019, 0028, 0031-0032, 0048, 0057. The system processes the received data to determine active speaker location. The system executes an active speaker location program and using the image data and three-dimensional model, the active speaker location program locates the second microphone array on the table in three dimensions. The three-dimensional location of an active speaker may be utilized by the active speaker location program to compute a setting for the color camera of the video conferencing device. In some examples, the setting may comprise one or more of an azimuth of the active speaker with respect to the color camera, an elevation of the active speaker with respect to the camera, and a zoom parameter of the camera. An ASD program of the active speaker location program uses color image data to identify a head and/or face of an active speaker. The video conferencing device may then highlight the active speaker by providing the zoomed-in video feed to the one or more other computing devices participating in the video conference, such as in an inset video window within a larger video window showing the room.).
Whyte discloses an active speaker program that is utilized to track active speakers location via camera and microphone data.
Aas discloses enables the data received from each speaker tracking system of a video conference endpoint to be dynamically and automatically utilized to continuously calibrate the multiple speaker tracking systems of a collaboration endpoint (see fig. 6, 9, col. 11, line 36-col. 12, line 38.).
The combination of Aas and Whyte provide the continuously calibrating the tracking system to track active speakers in a room.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte to incorporate the system continuously calibrating automatically to track the active speaker. The modification provides for programming in Whyte and Aas to be incorporated to calibrate continuously while tracking active speakers.
Regarding claim 6, Whyte teaches a method comprising: connecting a processing unit to a first camera and a first microphone in a first meeting room (see fig. 1, ¶ 0014-0015, 0018. The system having a microphone array that receives audio data from the room (14) , a camera that includes a color camera, such as an RGB camera, that captures color image data from the room (14).); generating, with a calibration module of the processing unit, a calibration strategy; obtaining, with the processing unit, video data from the first camera in accordance with the calibration strategy; translating, with the processing unit, the video data into spatial data; identifying, with a mapping module of the processing unit, a physical location of the first microphone in the first meeting room from the spatial data; and determining, with the calibration module, a field of view operating parameter of the first camera in response to the spatial data (see fig. 1, ¶ 0017-0019, 0028, 0031-0032, 0048, 0057. The system processes the received data to determine active speaker location. The system executes an active speaker location program and using the image data and three-dimensional model, the active speaker location program locates the second microphone array on the table in three dimensions. The three-dimensional location of an active speaker may be utilized by the active speaker location program to compute a setting for the color camera of the video conferencing device. In some examples, the setting may comprise one or more of an azimuth of the active speaker with respect to the color camera, an elevation of the active speaker with respect to the camera, and a zoom parameter of the camera. An ASD program of the active speaker location program uses color image data to identify a head and/or face of an active speaker. The video conferencing device may then highlight the active speaker by providing the zoomed-in video feed to the one or more other computing devices participating in the video conference, such as in an inset video window within a larger video window showing the room.).
Whyte discloses an active speaker program that is utilized to track active speakers location via camera and microphone data.
Aas discloses enables the data received from each speaker tracking system of a video conference endpoint to be dynamically and automatically utilized to continuously calibrate the multiple speakers tracking systems of a collaboration endpoint (see fig. 6, 9, col. 11, line 36-col. 12, line 38.).
The combination of Aas and Whyte provide the continuously calibrating the tracking system to track active speakers in a room.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte to incorporate the system continuously calibrating automatically to track the active speaker. The modification provides for programming in Whyte and Aas to be incorporated to calibrate continuously while tracking active speakers.
Regarding claim 14, Whyte teaches a method comprising: connecting a processing unit to a first camera and a first microphone, each of the first camera and first microphone located in a first meeting room; connecting the processing unit to a second camera and a second microphone, each of the second camera and second microphone located in a second meeting room; generating, with a calibration module of the processing unit, a room calibration strategy for the first meeting room and the second meeting room (see fig. 1, ¶ 0014-0015, 0018-0019, 0028. The system having a microphone array that receives audio data from the room (14) , a camera that includes a color camera, such as an RGB camera, that captures color image data from the room (14).). For meetings in a different room, the video conferencing device of the present disclosure may accurately determine the location of an active speaker in such rooms or other configurations.); conducting the room calibration strategy, with the processing unit, to identify visual characteristics and acoustic characteristics of different locations within each meeting room (see ¶ 0014-0015, 0018-0019, 0028, 0038. Image data from the image capture device(s) may be used by the active speaker location program to generate a three-dimensional model of at least a portion of the room. Image data also may be used to construct still images and/or video images of the surrounding environment from the perspective of the video conferencing device. The image data also may be used to measure physical parameters and to identify surfaces of a physical space, such as the room.); identifying, with a learning module of the processing unit, a first participant in the first meeting room and a second participant in the second meeting room (see fig. 1, ¶ 0028, 0042, 0050. ASD program of the active speaker location program utilizes data to estimate the location of the active speaker. An ASD program may initially select multiple potential active speakers, such as the second participant and third participant, based on the SSL distribution corresponding to the first audio data from the first microphone array. Using color image data from the RGB camera, for each potential active speaker the ASD program may determine the location of the person's head, as indicated by the bounding boxes.); assigning, with an identification module of the processing unit, a first unique identifier to the first participant and a second unique identifier to the second participant (see ¶ 0058. the active speaker may be highlighted in the video feed to the other computing device(s) by visually emphasizing the active speaker via, for example, an animated box or circle around the head of the active speaker, an arrow pointing to the active speaker, on-screen text adjacent to the active speaker (such as, “John in speaking”.); recognizing, with the processing unit, ambiguation in tracking the first participant (see ¶ 0061-0062. The active speaker location program determines that the first microphone array and/or the second microphone array has moved from a first location to a second, different location.); executing, with the processing unit, the room calibration strategy to alter an operating parameter of the first camera to disambiguate tracking of the first participant; obtaining, with the processing unit, video data from the first camera in accordance with the room calibration strategy (see ¶ 0057, 0061-0062. Based on determining that at least one of the first microphone array and the second microphone array has moved, the active speaker location program recomputes one or more of the location and the angular orientation of the second microphone array. In this manner, the active speaker location program updates the relative positions of the second microphone array and the video conferencing device to ensure continued accuracy of the estimated location of the active speaker.); translating, with the processing unit, the video data into spatial data; identifying, with a mapping module of the processing unit and from the spatial data, a physical location of the first microphone in the first meeting room and a physical location of the first participant in the first meeting room (see ¶ 0029, 0031. The video conferencing device captures image data of the room using one or both of the RGB camera and the depth camera. Using this image data, the active speaker location program of video conferencing device generates a three dimensional model of at least a portion of the room. Using the image data and the three dimensional model, the active speaker location program locates the second microphone array on the table in three dimensions relative to the image capture device(s) of the video conferencing device, such as the RGB camera and/or the depth camera.); determining, with the calibration module, a field of view operating parameter of the first camera in response to the spatial data; and adapting, with an adaptation module of the processing unit, at least one operating parameter of the first camera in response to the physical location of the first participant, the adaptation module adapting the at least one operating parameter with respect to the identified physical location of the first participant in the first meeting room (see fig. 1, ¶ 0017-0019, 0028, 0031-0032, 0048, 0057. The system processes the received data to determine active speaker location. The system executes an active speaker location program and using the image data and three-dimensional model, the active speaker location program locates the second microphone array on the table in three dimensions. The three-dimensional location of an active speaker may be utilized by the active speaker location program to compute a setting for the color camera of the video conferencing device. In some examples, the setting may comprise one or more of an azimuth of the active speaker with respect to the color camera, an elevation of the active speaker with respect to the camera, and a zoom parameter of the camera. An ASD program of the active speaker location program uses color image data to identify a head and/or face of an active speaker. The video conferencing device may then highlight the active speaker by providing the zoomed-in video feed to the one or more other computing devices participating in the video conference, such as in an inset video window within a larger video window showing the room.).
Whyte discloses an active speaker program that is utilized to track active speakers location via camera and microphone data.
Aas discloses enables the data received from each speaker tracking system of a video conference endpoint to be dynamically and automatically utilized to continuously calibrate the multiple speakers tracking systems of a collaboration endpoint (see fig. 6, 9, col. 11, line 36-col. 12, line 38.).
The combination of Aas and Whyte provide the continuously calibrating the tracking system to track active speakers in a room.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte to incorporate the system continuously calibrating automatically to track the active speaker. The modification provides for programming in Whyte and Aas to be incorporated to calibrate continuously while tracking active speakers.
3. Claim(s) 2, 3, 4, 5, 7, 8, 9, 10, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360).
Regarding claim 2, Whyte teaches the system of claim 1, wherein the spatial data comprises information describing an object located in the room (see ¶ 0031. Using the image data and the three-dimensional model, the active speaker location program locates the second microphone array on the table in three dimensions relative to the image capture device(s) of the video conferencing device, such as the RGB camera and/or the depth camera.).
Regarding claim 3, Whyte teaches the system of claim 1, wherein the processing unit comprises a processor and non- volatile memory (see ¶ 0017. The video conferencing device includes an active speaker location program that is stored in mass storage of the video conferencing device. The active speaker location program is loaded into memory and executed by a processor of the video conferencing device.).
Regarding claim 4, Whyte teaches the system of claim 3, wherein the processing unit is physically positioned within the room (see fig. 1, ¶ 0015, 0017. The active speaker location program is loaded into memory and executed by a processor of the video conferencing device.).
Regarding claim 5, Whyte teaches the system of any of claims 1, wherein the field of view operating parameter corresponds with an extent of the room observable by the visual sensor (see fig. 1-2, ¶ 0019-0021. Image data from the image capture device(s) may be used by the active speaker location program to generate a three-dimensional model of at least a portion of the room. Such image data also be used to construct still images and/or video images of the surrounding environment from the perspective of the video conferencing device. The image data also may be used to measure physical parameters and to identify surfaces of a physical space, such as the room. Surfaces of the room may be identified based on depth maps derived from color image data provided by the color camera. In other examples, surfaces of the room may be identified based on depth maps derived from depth image data provide by the depth camera.).
Regarding claim 7, Whyte teaches the method of claim 6, further comprising combining, with the processing unit, content from the first camera and the first microphone into a virtual meeting displayed in a second meeting room (see fig. 1, ¶ 0014-0016, 0020-0021. The information obtained in the first room is broadcasted to the second room or second devices.).
Regarding claim 8, Whyte teaches the method of claim 6, wherein the calibration strategy prescribes at least one test to obtain the video data pertinent to translating the video information into the spatial data (see ¶ 0031-0032, 0038. The system analyzes the data in order to determine location, this would provide testing or determining what data is performed in order to provide location data based on received data.).
Regarding claim 9, Whyte teaches the method of any of claims 6, wherein the video data is acquired from multiple separate cameras in the first meeting room (see ¶ 0029, 0031. The video conferencing device captures image data of the room using one or both of the RGB camera and the depth camera. Using this image data, the active speaker location program of video conferencing device generates a three-dimensional model of at least a portion of the room.).
Regarding claim 10, Whyte teaches the method of any of claims 6, wherein the physical location of the first microphone is unknown by the processing unit until the translating the video data into the spatial data (see ¶ 0031-0032. The system determines by using image data the location of the microphones. The microphone is unknown until the image data is used to determine location of said microphones.).
Regarding claim 11, Whyte teaches the method of any of claims 6, wherein the calibration strategy prescribes determining, from the spatial data, physical coordinates of the first microphone within the first meeting room (see ¶ 0019, 0029-0032. A three-dimensional model may comprise the surfaces and objects in front of the video conferencing device in the positive z-axis direction, including at least portions of the first participant, second participant and third participant. The three-dimensional model may be determined using a three dimensional coordinate system having an origin at the video conferencing device).
4. Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of CN109167998A, Chen Haibo (2018) Translation
Regarding claim 12, Whyte and Asa do not teach the method of any of claims 6, further comprising generating an artificial intelligence strategy with the processing unit, the artificial intelligence strategy prescribing at least one test to determine operational capabilities of the first camera.
Haibo teaches generating an artificial intelligence strategy with the processing unit, the artificial intelligence strategy prescribing at least one test to determine operational capabilities of the first camera (see ¶ 0030-0037. The neural network determines if the camera is available.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte and Asa to incorporate an AI system determine camera availability. The modification provides for camera state if camera can be used or not.
5. Claim(s) 13 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of CN109167998A, Chen Haibo (2018) Translation in further view of Feng et al. (US 2014/0049595).
Regarding claim 13, Whyte, Asa and Haibo do not teach the method of claim 12, further comprising correlating the operational capabilities of the first camera to different physical locations within the first meeting room.
Feng teaches comprising correlating the operational capabilities of the first camera to different physical locations within the first meeting room (see abstract, ¶ 0084, 0087, 0103. The location of the loudspeaker 119 may remain the same in the environment regardless of which participants are present, the video device 110 can recognize that this location corresponds to the loudspeaker 119 and not to a participant so that audio detection and pan angle estimation associated with the loudspeaker's location can be ignore, A single participant P1 present in the environment. Processing as described herein detects the face F1 of the participant P1 and determines the location of the face F1 in the adjunct camera's stationary view 300 of the environment. Based on this determined location, the main camera (150) is directed to capture a framed view 310 of the single participant P1.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa and Haibo to incorporate having a video device to recognize the location of affixed speaker and not consider said location as containing participant. The modification provides for reliable auto framing of participants in a video conference.
6. Claim(s) 15 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of Feng et al. (US 2014/0049595).
Regarding claim 15, Whyte and Aas do not teach the method of claim 14, further comprising eliminating, with the processing unit, at least one false positive to locate the first participant within the first meeting room.
Feng teaches eliminating, with the processing unit, at least one false positive to locate the first participant within the first meeting room (see Abstract, ¶ 0103, 0117. Automatic framing of participants in a videoconference environment, discussion now turns to further features of the present disclosure that enhance the auto-framing achieved. As can be appreciated, the detection results from the face detector 202 of FIG. 4 may not always be reliable when performing the auto-framing. For example, the face detector 202 can have false alarms or misses when the results are false positives and false negatives.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa to incorporate allowing false positives based false face detections. The modification provides for determining false positives of a participants.
7. Claim(s) 16, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of Pauik. (US 2024/0257816).
Regarding claim 16, Whyte and Aas do not teach the method of claim 14, further comprising loading, with the processing unit, a participant profile for the first participant in response to identifying the first participant in the first meeting room.
Pauik teaches loading, with the processing unit, a participant profile for the first participant in response to identifying the first participant in the first meeting room (see fig. 7, ¶ 0080. The system identifies the participant and displays a name associated with the participant.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa to incorporate presenting a name based on identity being looked up by a profile. The modification provides for determining the participants identity.
Regarding claim 17, Whyte teaches the method of claim 16, wherein the operating parameter of the first camera is adapted to provide accurate recording of activity of the first participant in response to the physical location of the first participant within the first meeting room relative to a physical location of the first camera in the first meeting room (see ¶ 0057. Active speaker may be utilized by the active speaker location program to compute a setting for the color camera of the video conferencing device. In some examples, the setting may comprise one or more of an azimuth of the active speaker with respect to the color camera, an elevation of the active speaker with respect to the camera, and a zoom parameter of the camera. In some examples, a video capture program may use the setting to highlight the active speaker. In one example and with reference to FIG. 2, a setting for the RGB camera may comprise an azimuth of the active speaker with respect to the camera (in the X-Z plane), an elevation of the active speaker with respect to the camera (in the Y direction), and a zoom parameter for the camera.).
8. Claim(s) 18 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of Rowley (US 2021/0233304).
Regarding claim 18, Whyte and Aas do not teach the method of any of claims 14, further comprising tracking the first participant over time with the first camera to a plurality of different coordinates within the first meeting room, wherein the plurality of different coordinates are computed by the processing unit.
Rowley teaches tracking the first participant over time with the first camera to a plurality of different coordinates within the first meeting room, wherein the plurality of different coordinates are computed by the processing unit (see fig. 1-2, ¶ 0112. The system tracks the user position and movements in aera. This would provide different coordinates for each participants movement in a room.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa to incorporate tracking a participant in a conferencing room. The modification provides for tracking a participant when moving to different locations.
9. Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of Bhatt (US 2025/0139968).
Regarding claim 19, Whyte and Aas do not teach the method of any of claims 14, wherein the room calibration strategy prescribes different operating parameters for the first camera and the first microphone for different physical locations of the first participant in the first meeting room.
Bhatt teaches wherein the room calibration strategy prescribes different operating parameters for the first camera and the first microphone for different physical locations of the first participant in the first meeting room (see fig. 6, ¶ 0041. An automatic calibration phase can be activated automatically when a first participant enters a FOV of the camera, as will be discussed below in greater detail. In addition, the automatic calibration phase can be activated for a pre-determined amount of time, e.g., 30 seconds, 60 seconds, 120 seconds, 300 seconds, etc., or the automatic calibration phase can be continuously active. The automatic calibration phase can track participant location in a conference room for a longer period of time, e.g., hours or days, to generate a predictable model of participant location in the conference room, meaning that an inclusion zone can be automatically updated or changed over time.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa to incorporate calibrating phase when a participant enters or moves in a conferencing room. The modification provides calibrating the system when a participant moves to different locations.
10. Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Whyte et al. (US 2017/0201825) in view of Aas et al. (US 9,986,360) in further view of O’ Leary et al. (US 2022/0247919).
Regarding claim 20, Whyte and Aas do not teach the method of any of claims 14, further comprising altering an operating parameter of the first camera proactively, with the processing unit, in response to behavior of the first participant predicted by the processing unit.
O’Leary teaches altering an operating parameter of the first camera proactively, with the processing unit, in response to behavior of the first participant predicted by the processing unit (see fig. 6L, ¶ 0250. When the system exhibits behavior during a conference the system can change or update the camera view in regard to participant behavior.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Whyte, Asa to incorporate update the camera based on participant behavior. The modification provides updating the camera parameters based on participant behavior.
Conclusion
11. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASSAD MOHAMMED whose telephone number is (571)270-7253. The examiner can normally be reached 9:00AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Duc Nguyen can be reached at 571-272-7503. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASSAD MOHAMMED/ Examiner, Art Unit 2691
/DUC NGUYEN/ Supervisory Patent Examiner, Art Unit 2691
Prosecution Insights