Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: Information Processing Apparatus, and Information Processing Method for Controlling Sound Image Localization.
Response to Arguments
Claims 1, 12, and 15-18 have been amended.
Claims 1-18 are presently pending.
Applicant’s arguments with respect to claims 1, 17, and 18 have been considered but are moot in view of the new ground(s) of rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 12-14, and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tsurumi (US 2012/0224043 A1) (of record, hereinafter Tsurumi), further in view of Strong (US 2012/0304216 A1) (hereinafter Strong), further in view of Eronen et al. (US 2018/0295463 A1) (of record, hereinafter Eronen), and further in view of Lemmey et al. (US 2013/0021431 A1) (of record, hereinafter Lemmey).
Regarding Claim 1, Tsurumi discloses an information processing apparatus [Figs. 1, 6] comprising
a Central Processing Unit (CPU) [Figs. 1, 6] configured to:
output sound control information [Figs. 1, 5; Audio output control unit 111] based on a basis of first analysis result and a second analysis result, wherein the first analysis result corresponds to first time-series data included in content data, the second analysis correspond to second time-series data, the second time-series data indicates a situation of a first user, [Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data); 0034, 0054-60: content data may be analyzed to detect and determine scenes and characteristics of scenes in the content (i.e., first time-series data); 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing state of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio] and
the sound control information includes second information to control a sound included in the content data. [Fig. 5; 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing state of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio]
analyze the second time-series data; [Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data)]
detect a viewing state of the first user based on the analysis of the second time-series data; [Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data)]
based on the detected viewing state generate the second information to control the sound included in the content data; and control the sound included in the content data based on the second information. [Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data); 0031, 0063-80: depending on viewing state of user, sound of video content may be adjusted/controlled]
Tsurumi fails to explicitly disclose based on the detected viewing state that indicates that a degree of excitement of the first user is higher than a reference, generate the second information to control the sound included in the content data; and control the sound included in the content data based on the second information.
Strong, in analogous art, teaches based on the detected viewing state that indicates that a degree of excitement of the first user is higher than a reference, generate the second information to control the sound included in the content data; and control the sound included in the content data based on the second information. [0008-10, 0042-45: sensors 108a may sense characteristics of user (such as the video/audio capturing inputs of Tsurumi above); 0023-24, 0046-50: if a user is sensed to be in a highly excited state, volume output of the audio portion may be controlled, where sensed characteristics may include a magnitude/change in magnitude threshold to conclude the user is in a highly excited state]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the apparatus of Tsurumi with the teachings of Strong to generate and control the sound of the content data based on detecting a high degree of excitement so as to automatically adjust presentation of content to maintain or enhance the user’s excited state. [Strong – 0014]
Tsurumi and Strong fail to explicitly disclose wherein the sound control information includes second information to control sound image localization of a sound included in the content data.
Eronen, in analogous art, teaches wherein the sound control information includes second information to control sound image localization of a sound included in the content data. [Figs. 1-7; 0102, 0121-127: audio source localization may be determined through visual analysis of video for tracking sound sources in the video (such as the first and second analysis results of Tsurumi above)]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the apparatus of Tsurumi and Strong with the teachings of Eronen to analyze video content to determine sound image localization in order to automate the process of spatial audio capture, mixing and sound track creation. [Eronen – 0002-4, 0102]
Tsurumi, Strong, and Eronen fail to explicitly disclose the sound control information includes first information to control sound image localization of a voice of a second user; and to output the voice of the second user to a user terminal associated with the first user.
Lemmey, in analogous art, teaches the sound control information includes first information to control sound image localization of a voice of a second user; and to output the voice of the second user to a user terminal associated with the first user. [Figs. 2-6; 0011, 0019-20, 0048-53: group ‘ensemble’ experience where video and audio from participants may be provided and output alongside presentation of some primary content, whereby sound of respective participants and base content may vary according to a variety of rules; 0056-57: where participants may be paced as objects within the video and be visually and audially adjusted to appear and sound further away/quieter etc.]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the apparatus of Tsurumi, Strong, and Eronen with the teachings of Lemmey to output voice of a second user to a user terminal of a first user in order to provide means for controlling attention management for ensemble experiences with video chatting elements and localize the voice data accordingly to mimic real world experiences. [Lemmey – ABST; 0011-12, 0057]
Regarding Claim 2, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi and Lemmey disclose wherein the CPU is further configured to [Tsurumi – Figs. 1, 6; 0087, 0090-91: connection ports 923/communication device 925] transmit one of the content data or the voice of the second user to the user terminal; transmit and the sound control information to the user terminal. [Tsurumi – Figs. 1, 5; 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing stat of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio; 0025: apparatus 100 provides content to display device 10, where content includes video data associated with content and audio data according to the output control unit; Lemmey – Figs. 2-6; 0011, 0019-20, 0048-53]
Regarding Claim 3, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi and Lemmey disclose wherein the CPU is further configured to [Tsurumi – Figs. 1, 6; 0087, 0090-91: connection ports 923/communication device 925] apply the sound control information to one of the sound included in the content data or the voice of the second user; output distribution data based on the application of the sound control information to the one of the sound included in the content data or the voice of the second user; and transmit the distribution data to the user terminal. [Tsurumi – Figs. 1, 5; 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing stat of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio; 0025: apparatus 100 provides content to display device 10, where content includes video data associated with content and audio data according to the output control unit; Lemmey – Figs. 2-6; 0011, 0019-20, 0048-53]
Regarding Claim 4, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi and Lemmey disclose wherein the sound control information further includes one of third information to control a volume of the voice of the second user output to the user terminal or fourth information to control a volume of the sound included in the content data. [Tsurumi – Figs. 1, 5; 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing stat of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio; Lemmey – Figs. 2-6; 0011, 0019-20, 0048-53]
Regarding Claim 5, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi discloses wherein the sound control information further includes one of fifth information to control sound quality of the voice of the second user output to the user terminal or sixth information to control a volume of the sound included in the content data. [Tsurumi – Figs. 1, 5; 0031-32, 0065, 0068, 0074, 0077, 0080: where depending on viewing stat of user and/or scene importance, audio output control unit 111 may control volume and/or quality of content audio]
Regarding Claim 12, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 2, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi wherein the viewing state includes one of information indicating occurrence of a conversation between the first user and the second user, information indicating occurrence of a reaction of the first user is making a reaction, or information indicating whether the first user watches a screen of the user terminal, and the CPU is further configured to output the sound control information based on the detected viewing state. [Tsurumi – Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data); 0064-68: where system may determine that the user’s eyes are close (i.e., not watching the screen) and may gradually lower volume of the audio as a result of that determination]
Regarding Claim 13, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 12, which are analyzed as previously discussed with respect to that claim.
As Claim 13 is directed toward an alternative limitation (viewing state including whether or not the user is having a conversation with another user), and another alternative (information indicating whether or not the user is watching a screen) was chosen for examination and application of art in Claim 12, the further limitations of the unselected alternative limitation will not be addressed and Claim 13 are rejected with Claim 12.
Regarding Claim 14, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 12, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi, Strong, Eronen, and Lemmey disclose wherein in a case where the detected viewing state indicates that the first user does not watch the screen of the user terminal, the CPU is further configured to: generate the second information to control the sound image localization of the sound included in the content data; and control the sound image localization of the sound included in the content data based on the second information, wherein based on the controlled sound image localization of the sound included in the content data the first user feels the sound included in the content data is heard from a farther place than a sound heard immediately before a time point at which the detected viewing state indicates that the first user does not watch the screen, and the sound image localization of the sound included in the content data is controlled until the detected viewing state indicates that the first user watches the screen. [Tsurumi – Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data); 0064-68: where system may determine that the user’s eyes are close (i.e., not watching the screen) and may gradually lower volume of the audio as a result of that determination (where it would be understood that a lower volume would make sounds feel further away); Eronen – Figs. 1-7; 0102, 0121-127; Lemmey – Figs. 2-6; 0011, 0019-20, 0048-53]
Regarding Claim 17, Claim 17 recites a method the performs the functions of the apparatus of Claim 1. As such, Claim 17 is analyzed and rejected similarly as Claim 1, mutatis mutandis.
Regarding Claim 18, Claim 18 recites a CRM the performs the functions of the apparatus of Claim 1. As such, Claim 18 is analyzed and rejected similarly as Claim 1, mutatis mutandis.
Claim(s) 6-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tsurumi, Strong, Eronen, and Lemmey as applied to claim 2 above, and further in view of Miyasato et al. (US 2004/0246259 A1) (of record, hereinafter Miyasato).
Regarding Claim 6, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi discloses wherein the CPU is further configured to: analyze the first time-series data; and detects scenes of a content based on the analysis of the first time-series data. [Tsurumi – 0034, 0054-60: content data may be analyzed to detect and determine scenes and characteristics of scenes in the content (i.e., first time-series data)]
Tsurumi, Strong, Eronen, and Lemmey fail to explicitly disclose wherein the CPU is further configured to: detect a progress status of a content . (Emphasis on the element of the limitation not explicitly disclosed by Tsurumi, Strong, Eronen, and Lemmey).
Miyasato, in analogous art, teaches wherein the CPU is further configured to: detect a progress status of a content . [Figs. 1, 5, 7; 0028-30, 0039-40, 0044-45: system may analyze video data to detect if a current section is a music/signing section of the video]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the apparatus of Tsurumi, Strong, Eronen, and Lemmey with the teachings of Miyasato to specify an analysis unit to detect a progress status of a content in order to allow for users to automatically determine various positions and structure of segments of a musical program. [Miyasato – 0004-7]
Regarding Claim 7, Tsurumi, Strong, Eronen, Lemmey, and Miyasato disclose all of the limitations of Claim 6, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi and Miyasato further disclose wherein the CPU is further configured to detect, as the progress status, one of during performance, during a performer's utterance, before start, after end, during an intermission, or during a break. [Tsurumi – 0034, 0054-60: content data may be analyzed to detect and determine scenes and characteristics of scenes in the; Miyasato – Figs. 1, 5, 7; 0028-30, 0039-40, 0044-45: system may analyze video data to detect if a current section is a music/signing section of the video]
Regarding Claim 8, Tsurumi, Strong, Eronen, Lemmey, and Miyasato disclose all of the limitations of Claim 6, which are analyzed as previously discussed with respect to that claim.
Furthermore, Miyasato further discloses wherein the CPU is further configured to recognize music played in the content based on the detected progress status as during performance. [Miyasato – Figs. 1, 5, 7; 0028-30, 0039-40, 0044-45: system may analyze video data to detect if a current section is a music/signing section of the video]
Regarding Claim 9, Tsurumi, Strong, Eronen, Lemmey, and Miyasato disclose all of the limitations of Claim 6, which are analyzed as previously discussed with respect to that claim.
Furthermore, Miyasato further discloses wherein the CPU is further configured to analyze the first time-series data based on auxiliary information to improve accuracy of analysis, and the auxiliary information includes information indicating a progress schedule of the content, information indicating a song order, or information regarding a production schedule. [Miyasato – 0031-32, 0043-45: chapter menu information may be provided with the video signal for helping to select detect time periods over which specific songs occur]
Regarding Claim 10, Tsurumi, Strong, Eronen, Lemmey, and Miyasato disclose all of the limitations of Claim 6, which are analyzed as previously discussed with respect to that claim.
Furthermore, Miyasato further discloses wherein the CPU is further configured to detect a tune of music played in the content. [Miyasato – Fig. 7; 0031, 0039-45: music/singing and title of song may be detected and identified]
Regarding Claim 11, Tsurumi, Strong, Eronen, Lemmey, and Miyasato disclose all of the limitations of Claim 6, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi and Eronen disclose wherein the first time-series data includes time-series data of a first video of the content, and the CPU is further configured to determine information of sound image localization corresponding to the time-series data of the first video of the content at a certain point of time, wherein the information of the sound image localization corresponding to the time-series date of the first video of the content is determined based on a model information, the model information is obtained based on a learning operation, and the learning operation is based on a second video of a state where one or more pieces of music are played and information of sound image localization of a sound corresponding to the second video. [Tsurumi – 0034, 0054-60: content data may be analyzed to detect and determine scenes and characteristics of scenes in the content (i.e., first time-series data); Eronen – Figs. 1-7; 0102, 0121-127: audio source localization may be determined through visual analysis of video for tracking sound sources in the video; 0232, 0243: neural network trained classifiers may be utilized for audio and visual space classifiers (where it would be implicitly understood that neural networks – NN – are trained with like data of the information to be classified/detected by the NN)]
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tsurumi, Strong, Eronen, and Lemmey as applied to claim 12 above, and further in view of Liu et al. (US 20170257669 A1) (of record, hereinafter Liu).
Regarding Claim 15, Tsurumi, Strong, Eronen, and Lemmey disclose all of the limitations of Claim 12, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tsurumi discloses wherein the second time-series data includes one of a voice of the first user, a third video of the first user, or information indicating an operation status of the user terminal of the first user. [Figs. 1-5; 0027-30, 0040-43, 0048-49: image and/or sound data of a user over time may be analyzed to determine a viewing state of the user (i.e., second time-series data)]
Tsurumi, Strong, Eronen, and Lemmey fail to explicitly disclose where the CPU is further configured to detect the degree of excitement of the first user based on at least one of the voice of the first user, the third video of the first user, or the information indicating the operation status.
Liu, in analogous art, teaches where the CPU is further configured to detect the degree of excitement of the first user based on at least one of the voice of the first user, the third video of the first user, or the information indicating the operation status. [0020-23: user engagement detector may use cameras and/or microphones (such as the image and sound data gathered by Tsurumi) to detect a level of excitement/engagement of a user by detecting the user’s facial expressions or based on analyzing their voice, and have varying different threshold for different emotions]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the apparatus of Tsurumi, Strong, Eronen, and Lemmey with the teachings of Liu to detect a degree of excitement of the user based on voice and video of the user as it would be readily understood that use of such sensor information may be utilized to determine how engaged a user is with displayed content based on their facial expressions or voice level/tone. [Liu – 0021]
Allowable Subject Matter
Claim 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM J KIM whose telephone number is (571)272-2767. The examiner can normally be reached 9:30am - 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hadi Armouche can be reached at (571) 270-3618. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM J KIM/Primary Examiner, Art Unit 2409
Prosecution Insights